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INTRODUCTION 



On February 3 through 7, 1968, the American Educational Research 
Association conducted a program of eleven training sessions for approx- 
imately 700 educational researchers prior to the Annual Meeting of the 
organization. The costs of the program were borne by AERA, the U. S. 
Office of Education, and the participants themselves. This document is 
a report to the funding agency and the officials of AERA concerning the 
research training presessions program. 
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HISTORY OF THE AERA PRESESSIONS PROGRAM 



The American Educational Research Association Research Training 
Presessions Program has evolved over the past four to five years to 
where it now occupies a prominent position among the activities of the 
organization. The Presessions Program grew out of informal meetings of 
one or two days duration involving only a few selected researchers in 
AERA with rather narrow, common interests in 1964 and 1965 to a compre- 
hensive program of eleven five-day sessions serving over 700 researchers 
(both members and non-members of AERA) and representing a total invest- 
ment by the U. S. Office of Education, institutions of higher education, 
and individuals of nearly a quarter million dollars in 1968. 

In 1964 and 1965, one or two small groups of researchers used the 
occasion of the Annual Meeting of AERA to meet and discuss their mutual 
interests. These meetings were not widely publicized and did not have the 
training of researchers as their function. They can, however, be regarded 
as the precursors of the AERA Research Training Presessions Program because 
it was in 1966 that the prototypal "Presession" was held as one of a group 
of three meetings in the tradition of these "special preconvention meetings." 
That one Presession that set a pattern subsequently adopted for the Presessions 
Program was the 1966 Presession on Experimental Design under the direction 
of Richard E. Schutz. The 1966 Presession on Experimental Design was full 
of "firsts". It was the first five-day Presession sponsored by AERA. It 
was the first formal research training program completed under Title IV 
of the Elementary and Secondary Education Act of 1965. It was the first 
Presession in connection with a professional meeting that was formally and 
fully evaluated with respect to the achievement of its objectives. 

Encouraged by the success and acceptance of the 1966 Presession on 
Experimental Design and the growing interest of researchers in the possi- 
bility of expanding and formalizing the other pre- convent ion meetings, AERA 
presented six courses as the 1967 Presessions Program under the direction of 
Richard E. Schutz. The sessions and their directors were as follows: 

1. Bayesian Statistical Analysis 

Donald Meyer, Syracuse University 

2. Curriculum Research and Evaluation 

Robert L. Baker, Arizona State University 

3. Design and Analysis of Comparative Experiments in Education 

Gene V. Glass, University of Illinois 

4. Educational Research Management Procedures 

Desmond Cook, Ohio State University 

5. Multivariate Design and Analysis in Educational Research 

Joe Ward, Southwest Educational Development Laboratory 
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6. Research Strategies with Culturally Deprived Children 
Martin Deutsch, New York University 

The 1967 Presessions Program was supported in part by a grant from 
the U. S. Office of Education under Title IV of ESEA 1965. Approximately 
500 researchers applied to the program, and 322 researchers who could be 
accommodated, actually participated. 

In February of 1967, Dr. John I. Goodlad, newly elected President of 
AERA appointed six persons to the AERA Presessions Committee: 

Gene V. Glass, Chairman 

Richard C. Lonsdale, Division A- Administration 

David B. Orr, Division B- Curriculum and Objectives 

Richard C. Cox, Division C - Learning and Instruction 

John A. Easley, Division D - Measurement and Research Methodology 

John 0. Crites, Division E- Student Development and Personnel 

Services 

This committee was charged with planning and conducting the 1968 Presessions 
Program. It first undertook to state the purpose of the AERA Presessions 
Program. The following statement adopted by the Committee is dated 
March 17, 1967: 

Statement of Purpose of the AERA Research 
Training Presessions 

Preamble: In February of 1966 and 1967 AERA conducted a 

limited program of Research Training Presessions prior to its 
Annual Meeting. In this program, groups of 40 to 75 educational 
researchers came together to receive instruction in research 
techniques from research methodologists. The fundamental purpose 
of this program was not always apparent to the directors and 
participants in these sessions nor to the general membership of 
AERA. A statement of purpose or intent of the AERA Research 
Training Program is overdue. This statement is a description of 
the service which AERA is extending to educational researchers 
through the medium of its Research Training Presessions Program. 

Supervised training in the technical skills used by the educational 
researcher is generally available to him only at considerable expense 
and inconvenience once he has completed his graduate education and 
has assumed full professional responsibilities. Often he must leave 
his work for an extended period of time and travel to find those 
competent to instruct him. Much of the expense and inconvenience 
can be spared him if instruction in research skills is condensed 
into short training sessions held either before or after the Annual 
Meeting of AERA. The purpose of the AERA Research Training Presessions 
Program is to train educational researchers in fundamental research 
skills, e.g., experimental design, statistical analysis, survey 
techniques, measurement theory and technique, electronic data 






processing, the functions of the computer in research, research 
management. The Research Training Presessions are intended to be 
instructional or disseminative of established research techniques s/ 
as opposed to generative of new substantive problems or directions 
for research in some particular area. The latter function is 
considered to be the purpose of symposia and conferences and, hence, 
it falls within the scope of the Annual Meeting of AERA and the 
activities of other professional organizations. It is also not the 
purpose of the Research Training Presessions to disseminate innovations 
in education (e.g., team teaching, the Initial Teaching Alphabet, 
micro- teaching) which are not properly research skills and techniques 
common to a large class of research activities. 

Preference for participation in any Research Training Presessions 
will be given to researchers who hold a doctorate. This decision was 
made on the assumption (gratuitous, perhaps) that persons not holding 
a doctorate still have ample opporunities to improve their research 
skills while pursuing an advanced degree. On the other hand, the 
character of graduate education makes it relatively inaccessible to 
persons holding a doctorate who have assumed full professional 
responsibilities. Moreover, AERA is sensitive to the issue of 
intrusion upon the domain of univerisities, viz., the domain of 
graduate education of researchers. 

It would be contrary to the spirit of this statement if the 
expressed purpose of the Research Training Presessions Program was 
regarded as fixed and immutable. The purpose of this program can 
properly be said to have evolved from the activities of the first few 
years of operations. We have simply attempted to identify that purpose 
so that AERA can see more clearly the need it is trying to meet and 
which needs remain unmet. It is hoped that in the future the Research 
Training Presessions Program will be altered when necessary and 
appropriate to meet the needs of the educational researcher. 

AERA Research Training Presessions Committee 
Gene V. Glass, Chairman 
Richard C. Lonsdale, Division A 
John A. Easley, Jr. Division B 
David B. Orr, Division C 
Richard C. Cox, Division D 



Guided by the above statement of purpose of the program, the Committee 
undertook the planning and execution of the 1968 Research Training Presessions 
Program. The record of these activities constitutes the remainder of this 
report. 
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EVALUATION OF LONG-RANGE EFFECTS OF A 
1967 RESEARCH TRAINING PRESF.SSION 



The evaluation of the 1968 Presessions Program necessarily deals 
with short - term effects* Some indication of the long - term effects of 
the AERA Presessions Program are given in this section. 

A brief questionnaire was mailed to the 70 pa-' icipants of the 
12£Z Presession on the Design and Analysis of Comparative Experiments. 

This Presession was held at Grossinger, New York, on February 3-7, 1967, 
and was under the direction of Gene V. Glass, Kenneth D. Hopkins, ’and 
Jason Millman. Thus, the questionnaire sought to assess the effects of one 
1967 Pre session more than one year later. A total of 47 of the 70 
participants responded to the questionnaire, producing a fairly typical 

return rate of 67%. The results are reported on the questionnaire itself 
in Table B. 



Table B 

AERA 1967 Presession Questionnaire 
Follow-up Evaluation 

Responses of the 47 respondents are recorded below: 

1. a. Have you made an attempt in the last 15 months to increase your 
knowledge of experimental design and analysis as a result of 
your attendance at the 1967 Presession (i.e., did you do something 
you might not have done if you hadn't attended the Presession?)? 

Yes - 41 No - 2 (Circle one) 

No response - 4 



b. If "Yes," how? 
freq . 

36 1. By studying the instructional materials handed out at the 

Presession. 

35 2. By independent study from textbooks. 

JL 3. By enrolling in a formal course. 

6 4. By attending one of the 1968 Presessions. 

v J5 5. By attending some other "short- course" or "workshop". 

5 6. Other. Please specify: 1. Conducting a 1968 Presession, 

2. Teaching experimental design, 3. Writing a book, 4. 
Designing a new course. 

2. a. Can you point to some specific use you have made of the skills and 
knowledge acquired at the 1967 Presession? 

Yes - 46 No - 0 (Circle one) 

No response - 1 
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b. 



If "Yes", what use(s)? 
freq . 



31 1. 

25 2. 

25 3. 

20 4. 

36 5. 

6 . 



In the design or analysis of research performed by me. 
In consulting with colleagues. 

In consulting with others on the design and analysis of 
experiments. 

In teaching my classes. 

In advising graduate students engaged in research. 
Other. Please specify: 



3. a. Have you felt more competent to read the research literature in 
your research specialty over the past 15 months than before ajs 
a result of the 1967 Presession? 



Yes - 40 No - 7 (Circle one) 



b. Have you felt more competent to design and analyze experiments 
over the past 15 months as a result of the 1967 Presessions ? 

Yes - 46 No - 1 (Circle one) 

4. a. Have you written a research paper-either published or unpublished 
which benefited from your attendance at the 1967 Presession? 



Yes - 27 No - 19 (Circle one) 

No response - 1 

1. If "Yes", and if the paper or papers were published, where 
were they or will they be published? J. Ed. Psych. - 3; 
USOE Report - 2; Amer. J. Ment. Def. - 2; J. Ed. Res. - 2; 
Reading Teacher; Child Devel.; J. Hum. Res.; Psych, in 
Schools; J. Creat. Bev.; Read. Res. Quar.; EPIE Forum. 



5. a. Have the conditions of your employment changed wholly or partially 
as a result of your attendance at the 1967 Presession? 

Yes - 7 No - 39 (Circle one) 

No response - 1 

If "Yes", please explain: was made Res. Dir. at SUNY; increase in 
teaching exper. des.; directs research of graduate students. 

b. Have you taken a moia active interest in some professional 
organization (e.g., AERA, ASCD) as a result of the 1967 
Presession? 



Yes - 16 No - 31 (Circle one) 

If "Yes", which organization: AERA named 14 times. Other 

organizations named were PDK, Amer. Voc. ED. Res. Assoc., ASCD, 
Can. Ed. Res. Assoc. 
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P lease rank each of the following activities from 1 (most valuable) 
to 5 (least valuable) in terms of the value of the activities for 
your professional growth: 



Average Rank 



Activity 



1.71 
3.02 
2.69 
3.87 

3.71 



A post-doctoral fellowship for a year of study. 
An AERA Presession like the one you attended. 

An eight-week ,f summer institute". 

Self-study from textbooks. 

A semester-long academic course. 



The response' to item #1 indicate that in almost all cases (41 out of 
47) the 1967 Presession acted as a stimulus to further study. Study of the 
subject matter did not cease with the end of the five-day sessions but 
was extended through rereading instructional materials obtained at the 
Presession and through self-study of textbooks. From the wording of the 
question, we may assume that these extended efforts at self-improvement can 
be attributed to attendance at the 1967 Presession. 



We see in item #2 that the skills acquired during the 1967 Presession 
were appl' Q d in educational research endeavors. About 80% of the respondents 
reported that the Presession helped them in advising graduate students 
engaged in research. About 60% of the respondents reported using the skills 
they acquired in designing and analyzing their own research. More than 
half of the respondents were helped in consulting with their colleagues 
and others on research design and analysis. Somewhat less than half of the 
respondents made use of the newly- acquired skills in their teaching. A 
"spread of effect" of instruction is evident in the respond is to item #2. 
Indeed, it is probably no exaggeration to say that literally hundreds of 
persons (students, faculty members, public school personnel, etc.) benefited- 
to a greater or lesser extent-from the instruction given to 70 participants 
in the 1967 Design and Analysis Presession. 

In item #3, it is seen clearly that in the opinion of the respondents, 
participation in the 1967 Presession resulted in increased research competence 
which was not transitory, but was maintained 15 months after the Presession. 

In item #4, about 60% of the respondents indicated that the skills 
acquired in the 1967 Presession were put to use in reporting published or 
unpublished research. As can be seen under 4(a), an impressive array of 
professional journals are the benefactors of instruction at the 1967 Design 
and Analysis Presession. 

Seven out of 47 persons indicated in item #5 that the conditions of 
theii employment were changed as a result of attendance at the 1967 
Presession. In all but one instance, the "change" was one of emphasis 
and responsibility at the participant's previous place of employment instead 
of a change of place of employment. In part(b) of #5, 14 of 47 persons 
indicated that they have taken a greater interest in AERA as a result of 
attendance at the 1967 Presession. 



Item #6 is of particular interest. Respondents were asked to rank 
five educational activities from 1 (most valuable) to 5 (least valuable) 
in terms of their professional growth. Average ranks were calculcated 
for the rankings of the 47 respondents. The lower the average rank, the 
more valuable the activity was considered to be by the group of respondents . 
Attendance at an AERA Presession was ranked third among the five activities. 

The respondents considered the Presession more valuable than taking a semester - 
long academic course of self-study from textbooks. Indeed, attendance at 
a Presession was considered only slightly less valuable than attendance at 
an eight-week long "summer institute". 

Conclusion : The effects of a 1967 Presession were maintained over 15 

months, were spread to the colleagues and students of the part icipants, 
and were considered only slightly less valuable than those which might 
result from attendance at an eight-week "summer institute" . 
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SELECTING THE PRESESSIONS PROGRAM 



In March 1967, proposals for research training sessions were solicited 
by two means: (1) an announcement appeared in the March issue of the 

Educational Researcher , the official newsletter of AERA, that any individual 
was welcome to submit a proposal to conduct a presession; (2) invitations 
to conduct a presession were sent to some 200 leading researchers suggested 
by members of the Committee. Proposals were required to reach the Committee 
Chairman by April 1, 1967. 

Thirty proposals to conduct presessions were received by the deadline 
for submission. (Unfortunately, one excellent proposal was received after 
the Presessions Committee met to select the 1968 Program and could not be 
considered.) The Presessions Committee met in early April of 1968 in 
Chicago to evaluate the proposals and select the 1968 Presessions Program. 
Two of the proposals were screened out by the Chairman of the Committee 
prior to the meeting as not being worth the Committee’s time. The remaining 
28 proposals were read and evaluated by the Committee members in accordance 
with the schedule under I below and with respect to the criteria in II: 



Proposal Reading Schedule 



I. Schedule of Reading and Judging Proposals 
Division A: Lonsdale 

Proposals #1,3,4,7,8,11,13,14,19,21, and 27 



Division B: 
Proposals 



Glass 

#3,9,10,12,15,16,18,21,22,23,25, 



and 28 



Division C: 
Proposals 



Wiley 

#2,5,8,14,18,20,22,24,25,26, and 28 



Division D: 
Proposals 



Cox 

#2,6,9,10,11,12,15,17,19,20, and 26 



Division E: 
Proposals 



Thoresen 

#1,4,5,6,7,13,16,17,23,24, and 27 



(Each proposal is to be read by at least two persons.) 



II. Criteria for Rating Proposals 

A. Staff 

1. Experience and capability (low-high) 

2. Staff /participant ratio (low-high) 

3. Degree of commitment (low-high) 

B. Content (topic) 

4. Importance (need) (low-high) 

5. Appropriateness to Presession format (low-high) 

6. Extent of Planning 
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C. Potential Audience 

7. Probable size (small-large) 

8. Relationship to AERA (distant-close) 

D. Schedule 

9. Fullness (low-high) 

E. Evaluation 

10. Extent of planning (low-high) 

11. Comprehensiveness (low-high) 

(Each of the eleven criteria were rated on a scale from 0 to 9.) 

In addition, each proposal was given one of the following over-all 
ratings: 

A - Reject 

B - Accept conditionally 
C - Accept with recommendations 
D - Accept unconditionally 

Presession Proposals 

1. Astin, Alexander W. 

Assessing the Educational Environment and its Impact on Student 
Development 

2. Baker, Robert L. 

Planning Educational Experiments 

3. Bel lack, Amo 

Curriculum Theory 

4. Bolton, Dale L. 

Experimentation in Educational Administration 

5. Bormuth, John R. 

Research in Reading Instruction 

6. Campbell, David P. 

Interest Measurement Research 

7. Cook, Desmond 

Educational Research Management Procedures 

8. Cooper, James M. 

A Behavioral Approach to Teacher Education 

9. Gordon, Jack 

Application of Facet Analysis to Theory Construction 
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10. Hunka, S. M. 

APL: A Programming Language 

11. Lutz, Frank W. 

Anthropological Field Methodology in the Study of Education: With 

Particular Emphasis on Classroom Behavior and School Administration 

12. Marascuilo, Leonard A. 

Nonparametric Methods in Educational Research 

13. Marshall, Jon C. 

Practicum in Research Financing 

14. Mayo, Samuel T. 

Subdoctoral Training of Educational Research Workers 

15. Mayo, Samuel T. 

Mastery, Transfer, and Growth of Measurement Competency for 
Educational Personnel 

16. Medley, Donald M. 

Techniques for Measuring Teachers' Classroom Behavior 

17. Millman, Jason 

Design and Analysis of Comparative Experiments 

18. Naumann, Theodor F. 

Evaluation in Early Childhood Education 

19. Pace, C. Robert 

Evaluation: New Concepts in Scope, Strategy, and Purposes 

20. Page, Ellis B. 

The Computer and Natural Language 

21. Popham, W. James 

Instructional Product Research 

22. Ragsdale, Ronald G. 

On-line Computer Applications in Educational Research 

23. Romberg, Thomas A. 

Evaluating School Mathematics Programs 

24. Spencer, Richard E. 

Instructional Research 

25. Twelker, Paul A. 

Instructional Gaming and Simulation 

26. Ward, Joe H. _ 

Multivariate Design and Analysis in Educational Research 
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27. Warren, Jonathan R. , . 

Developmental Processes in College Students 

28 * ^Workshop on Strategies for Teaching and Evaluating Problem 
Solving Behaviors 

Those proposals which both readers voted "reject" were eliminated 
from further consideration. Those rated "accept unconditionally y 
both readers were selected for the Presessions Program. Proposals rat 
as "reject" by one reader and "accept" by the second reader were discussed 
by the Committee in light of the eleven evaluative criteria until x 

consensus (accept or reject) was reached. In certain instances, p p 
was read by two other Committee members who also voted acceptance or 
rejection. Eventually, eleven Presessions were judged worthy of support 
£y J the ^sessions Co^ittee. Their titles and directors were as follows. 

1. Research in Reading Instruction - John R. Bormuth, University of 
Chicago 

2. Educational Research Management Procedures - Desmond L. Cook, 

Ohio State University 

3 Anthropological Field Methodology in the Study of Education: 

With Particular Emphasis on Classroom Behavior and School 
Administration - Frank W. Lutz, New fork University 

4. Nonparametric Methods in Educational Research - Leonard A. Marascuilo, 
University of California-Berkeley 

5. Design and Analysis of Comparative Experiments - Jason Millman, 
Cornell University 

6. Evaluation: New Concepts in Scope, Strategy and Purposes 

C. Robert Pace, University of California - os nge e 

7. The Computer and Natural Language - Ellis B. Page, University of 
Connecticut 

R Instructional Product Research - W. James Popham, University of 

8 - California"- Los Angeles, and Howard Sullivan, Southwest Regrona 
Laboratory for Educational Research and Development - Los Angel 

11. Developmental Processes in College Students - Jonothan R. Warren, 
Educational Testing Service, Berkeley, California 
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PRESKSSIONS PLANNING MEETING 



On September 15, 1967, a meeting of the eleven Presessions Directors 
was held to discuss problems of publicity, application procedures, evaluation 
and coordinating administrative efforts. The minutes of this meeting 
follows: 



"Minutes of the AERA 1968 Research 
Training Presessions Directors Meeting" 



Date: Friday, September 15, 1967 

Place: O' Hare Inn, Chicago 

In Attendance: Presessions Directors Cook, Marascuilo, Millman, Page, 

Popham, Ragsdale, Ward, and Warren; Presessions Committee 
Chairman Glass, AERA Central Office representative Hanna, 
NISEC representative Beggs, Janos Koplyay 

1. Presession Sites: Sites have been chosen for all eleven Presessions: 

1. Bormuth - The Abbey on Lake Geneva 

2. Cook - Chicago-Sheraton 

3. Lutz - Chicago-Sheraton 

4. Marascuilo - Chicago-Sheraton 

5. Millman - The Abbey on Lake Geneva 

6. Pace - Chicago-Sheraton 

7. Page - Pheasant Run, St. Charles, Illinois 

8. Popham & Sullivan - Chicago-Sheraton 

9. Ragsdale - Pheasant Run, St. Charles, Illinois 

10. Ward - Northwestern University 

11. Warren - Oakton Manor, Pewaukee, Wisconsin 

2. Directors’ Names and Addresses: At Jim lupham' s suggestion, a list 

of the names and addresses of the Presessions Directors is enclosed 
in this mailing. 

3. Publicity: Notice of the 1968 Presessions should appear soon in 

issues of the American Psychologist , .Kappan (PDK) , APA Div. 15 Newsletter 
and the NCME Newsletter. It is probably too late to enter notices into 
other journals or periodicals. 

We will have to depend upon publicity (synopses of all 11 Presessions 
and an application form) in the October issue of the Educational 
Researcher which has already been mailed to the membership of AERA. 

In addition, a printed flier will soon be completed and sent to 
schools of education, and departments of psychology and sociology 
across the country. If you want copies of this flier for mailing 
to special sources, request as many as you need from Gene Glass. 
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Directors have complete freedom to publicize their individual Presessions 
where and how they wish. 

4. Application Procedures: The following procedures for processing 

applications have been adopted. 

A. Applications received by Glass by November 15. (A first 
and second choice of a Presession will be indicated on each 
application- see the October Educational Researcher. ) 

B. All applications indicating your Presession as first choice 
will be forwarded to you by November 18 . (You will continue 
to receive late applications until you inform me that you 
have started to select participants.) 

You will also receive a tally sheet on which the first and 
second choices of all applicants are recorded. 

C. Select "accepted" and "rejected" from among first choice 
applicants. (Take into consideration how many applicants 
chose your session second and the maximum number of partici- 
pants being accepted by the other sessions.) 

D. Return applications of "rejected" applicants and facsimiles 
of the applications of "accepted" applicants to Glass by 
December 2. (Be sure to indicate which were accepted and 
which were rejected.) 

E. By December 7 , you will receive the applications of all 
those who chose your session second-provided they were not 
accepted by their first-choice session and provided vacancies 
remain in your session. 

F. Select "accepted" and "rejected" from among your second-choice 
applicants and return applications (or facsimiles) to Glass. 

G. Notify all applicants of your decision by December 15 . 

(Notifying all applicants is your responsibility. However, 
if you have no room for any second-choice applicants, Glass 
will not send you second-choice applications, and he will 
notify them that your session is full.) 

5. Evaluation: The following information will be needed from each 

Director so that the Presessions Committee can produce a final 
evaluation report: 

a. Names of participants in attendance. 

b. Statement of objectives. 

c. A listing and two copies of all instructional and evaluation 
materials . 
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d. Summaries of results of mastery tests, attitude inventories, 
semantic differentials, etc. 

e. Actual schedule of activities. 

f. Attendance record (number in attendance only), 

g. Results of staff and participant critique forms. 

h. Director's written observations. 

The "staff" and "participant critique forms" will be prepared centrally 
and mailed to the Presession Directors in October. 

The evaluation reports of the 1966 and 1967 Presessions on the Design 
and Analysis of Experiments can be helpful to you in planning to 
evaluate your session. ' 

The planning meeting proved to be invaluable. It is recommended that 
similar meetings be included in future Presessions Programs . 



PUBLICITY 



Two main communications channels were utilized to publicize the 
1968 Presessions Program: mailings to academic departments, announce- 

ments in professional journals and newsletters. 

The yellow, printed announcement bound in this report was sent to 
all colleges of education listed in the Education Directory (Higher 
Education, Part 3, 1965-66) on September 1, 1967. Hence, almost every 
college of education in the country received an announcement of the 
Presessions Program which they were requested to post. 

Several copies of the same announcement were sent to the chief state 
school officer of every state in the United States plus Guam, Puerto Rico, 
the Trust Territory of the Pacific Islands, and the Virgin Islands on 
October 1, 1967. 

The same printed announcement was sent to 400 persons on the roles 
of the Association for Institutional Research on October 1, 1967. 

The following announcement appeared in the October 1967 issue of the 
Phi Delta KAPPAN , the official journal of Phi Delta Kappa, with a circulation 
of approximately 75,000 persons: 

"Training Sessions at AERA 

The American Educational Research Association has 
announced that 11 research training sessions will be 
held in Chicago, February 3 to 7, immediately preceding 
the annual AERA meeting. The sessions are open to any 
holder of a Ph.D. or Ed.D. degree, whether or not he 
is a member of AERA. Titles of the sessions and names of 
the directors are as follows: 

1. Research in Reading Instruction- John R. Bormuth, 

University of Minnesota. 

2. Educational Research Management Procedures 
- Desmond Cook, Ohio State University. 

3. Anthropological Field Methodology in the Study of 

Education: With Particular Emphasis on Classroom 

Behavior and School Administration-Frank W. Lutz, . 

New York University. 

4. Nonparametric Methods in Educational Resear ch- 

Leonard A. Marascuilo, University of California— 

Berkeley 
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5. Design and Analysis of Comparative Experiment s- 

> >' Jason Millman, Cornell University. 

6. Evaluation: New Concepts in Scope, Strategy, and 

Purposes-C. Robert Pace, University of California- 
Los Angeles. 

7. The Computer and Natural Language-Ellis B. Page, 
University of Connecticut. 

8. Instructional Product Research-W. James Popham, 
University of California-Los Angeles, and Howard Sullivan, 
Southwest Educational Research Lab-Los Angeles. 

9. On-line Computer Applications in Educational Research - 
G. Ronald Ragsdale, Ontario Institute for Studies in 
Education. 

10. Multivariate Design and Analysis in Educational Research 
Joe H. Ward, Southwest Educational Development Laboratory, 

San Antonio, Texas. 

11. Development Processes in College Students- Jonathan 

R. Warren, College Student Personnel Institute, Claremont, 
California. " 



The following announcement appeared in the September 1967 issue of 
the American Psychologist , a publication of the American Psychological 
Association with a circulation over 30,000: 

"At the Annual Meeting of the American Educational 
Research Association to be held in Chicago February 8-10, 

1968, 11 5-day sessions will be held as presessions. 

Participation in this program will not be restricted to 
AERA members, but is intended for persons who have an 
interest in educational research. Participation is 
restricted to persons holding a doctorate. For further 
information and application forms write to: Gene V. Glass, 

Laboratory of Educational Research, University of Colorado, 

Boulder, Colorado 80302." 

An announcement similar to the one mailed to all colleges of education 
appeared in the September issue of the NCME (National Council on Measurement 
in Education) Newsletter which has a circulation of approximately 2,000. 

The following announcement appeared in the newsletter of the National 
Society for Programmed Instruction: 

"November 15 is the registration deadline for the eleven 
five-day Research Training Presessions to be held by the American 
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Educational Research Association in Chicago from February 8-10, 

1968, just prior to its annual meeting there. 

These sessions will be conducted by outstanding research 
authorities from all parts of the United States, inclu g 
on "Instructional Product Research" with NSP mem er r. • 

James Popham of the UCLA Education Department and Howard Sul 
of the Southwest Educational Research Laboratory m os n 
as joint directors. 

No fees will be charged and participation in these Fresessions 
will not be restricted to AERA members. 

1 Further information can be obtained from Dr. Gene J^Boulder 

Laboratory of Educational Research, Universrty of Colordao, Boulder, 

Colorado 80302. " 

in several i^suef °of ’ 

This newsletter has a circulation of about 7,500 persons. 

A conservative estimate of the number of P^sons reached by the 
total effort to publicize the sessions Program 

^ganizations^suc^as^the o^schoo^Atoinistrators^th^ 1 * 

among public school personnel. 
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PLEASE POST 

AMERICAN EDUCATIONAL RESEARCH ASSOCIATION 
1968 RESEARCH TRAINING PRESESSIONS PROGRAM 



February 3-7, 1968 



A grant from the Training Research Branch of the U.S. Office of Education will make possible a 
program of eleven five-day sessions to be held as presessions in connection with the Annual Meeting of 
the American Educational Research Association in Chicago during February, 1968. The titles and 
names of the directors of the eleven research training sessions are as follows: 

1. Research iri Reading Instruction - John R. Bormuth, University of Chicago 

2. Educational Research Management Procedures - Desmond L. Cook, Chio 
State University 

3. Anthropological Field Methodology in the Study of Education: With Par- 
ticular Emphasis on Classroom Behavior and School Administration - Frank 
W. Lutz, New York University 

4. Nonparametric Methods in Educational Research - Leonard A. Marascuilo, 
University of California-Berkeley 

5. Design and Analysis of Comparative Experiments - Jason Millman, Cornell 
University 

6. Evaluation: New Concepts in Scope, Strategy and Purposes - C. Robert 
Pace, University of California-Los Angeles 

7. The Computer and Natural Language - Ellis B. Page, University of Con- 
necticut 

8. Instructional Product Research - W. James Popham, University of California- 
Los Angeles, and Howard Sullivan, Southwest Regional Laboratory for Edu- 
cational Research and Development-Los Angeles 

9. On-line Computer Applications in Educational Research - Ronald G. Rags- 
dale, Ontario Institute for Studies in Education 

10. Multivariate Design and Analysis in Educational Research - Joe H. Ward, 
Southwest Educational Development Laboratory, San Antonio, Texas 

I I. Developmental Processes in College Students - Jonathan R. Warren, Educa- 
tional Testing Service, Berkeley, California 

Participation in the AERA 1968 Research Training Presessions Program is not restricted to AERA 
members. The Presessions Program is intended for persons who have an interest in educational re- 
search. Participation is generally restricted to persons holding a doctorate — Ph.D. or Ed.D without re- 
gard to academic area. Neither fees nor tuition is charged for any of the sessions; however, each par- 
ticipant must pay his own board and room. 

Further information — synopses of the content of the sessions, etc. — and an application form ap- 
pear in the October issue of the AERA newsletter, Educational Researcher, or can be obtained by 
writing 

Dr. Gene V Glass 

Laboratory of Educational Research 

University of Colorado 

» i Boulder, Colorado 80302 

THE DEADLINE FOR RECEIPT OF APPLICATION FORMS IS NOVEMBER 15, 1967 
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APPLICATION PROCEDURES AND THE APPLICANTS 



The October 1967 issue of the Educational Researcher , the AERA news- 
letter, carried a "call for applications" to the 1968 Presessions Program. 
This announcement- -a copy of which is bound in this report- -carried a 
description of the staff, content and objectives, and the anticipated 
audience for each of the eleven Presessions; the requirements for 
application; and an application form. 

Each applicant completed the application form in part III of the 
announcement and sent it to the Director of the Presessions Program. 
Applications were received by the Director so their rate of receipt and 
distribution among the Presessions could be recorded. Although a dead- 
line of November 15, 1967, had been set for receipt of applications, too 
few applications were received by that date so the deadline was extended. 

A notice of the indefinite extension of the deadline appeared in the 
December issue of the Educational Researcher . Applications were received 
until January 31, 1968. With this alteration, then, the procedures for 
selecting the applicants was the one agreed upon at the September Directors 
meeting: 

1. Select "accepted" and"rejected" from among first-choice applicants. 
(Take into consideration how many applicants chose your session 
second and the maximum number of participants being accepted by 
the other sessions.) 

2. Return applications of "rejected" applicants and facsimiles of the 
applications of "accepted" applicants to Glass by December 2. (Be 
sure to indicate which were accepted and which were rejected.) 

3. By December 7 , you will receive the applications of all those who 
chose your session second--provided they were not accepted by their 
first- choice session and provided vacancies remain in your session. 

4. Select "accepted" and "rejected" from among your second-choice 
applicants and return applications (or facsimiles) to Glass. 

5. Notify all applicants of your decision by December 15. (Notifying all 
applicants is your responsibility. However, if you have no room 

for any second-choice applicants, Glass will not send you second- 
choice applications, and he will notify them that your session is 
full.) 

Most Presession Directors selected applicants until late January 1968. 

The rate of receipt of applications for the entire Presessions Program was 
as follows: 



Date 



Total Accumulated Number 
of Applicants 




a i 



September 22, 1967 
September 29, 1967 
October 6, 1967 
October 13, 1967 



79 

188 

236 
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October 20, 1967 


308 


October 27, 1967 


353 


November 3, 1967 


422 


November 10, 1967 


486 


November 17, 1967 


611 


November 24, 1967 


■» 


November 30, 1967 


■tt 


December 8, 1967 


670 


December 15, 1967 


a* 


December 22, 1967 


_ 


December 30, 1967 


712 


January 7, 1968 




January 15, 1968 


736 


January 22, 1968 


mm 


January 31, 1968 


743 



A graph of the cumulative number of applications over time is reproduced below: 




In Table A. appears the distribution of the applicants' first and 
second choices for Presessions. As examples of how Table A is read, 
six persons indicated session #1 as their first choice and session #3 
as their second choice, ten persons who indicated session #7 as their 
first choice did not indicate a second choice, and a total of 74 persons 
chose session #10 first. In the column farthest to the right appears the 
maximum number of participants which the Director would accept. The total 
maximum number of persons which could be accommodated in the total program 
was 745. 

It is interesting to note that about 40% of the persons who chose 
Presession #1 did not indicate a second choice; approximately 457 0 of those 
choosing Presession #3 first did not indicate a second choice. These 
two sessions (Reading Research and Anthropological Field Methodology) 
appear to have appealed to two groups of researchers with specific needs 
which could not be met by other Presessions on the program. Certain pairs 
of Presessions on the program are paired relatively often as either a first 
or second choice, indicating a clustering of sessions in terms of common 
interests of the applicants. Sessions #4 and #5 were often paired as first 
or second choices; sessions #6 and #11 (Warren) were frequently paired; 
sessions #7 and #9 (both involving computers) were often chosen together. 

The Presessions were ordered as follows in terns of number of first 
choices: 



Presession 



Number of Applicants 



Evaluation (6) 136 
Experimental Design (5) 82 
Reading Research (1) 76 
Anthropological Field Methodology (3) 76 
Multivariate Design and Analysis (11) 74 
Instructional Product Research (8) 69 
Research Management Procedures (2) 59 
Developmental Processes of College Students (10) 54 
Nonparametric Methods in Educational Research (4) 45 
On-line Computer Applications (9) 38 
Computer and Natural Language (7) 34 

743 



The large number of applicants to Presession #6 on Evaluation may be 
indicative of its broad appeal to practitioners and the great need for such 
instruction occasioned by federal legislation. Although the content of 
Presession #5 on Experimental Design was rather technical, its relatively 
great appeal may be due to the fact that the subject matter has application 
across most fields in educational research and that 1968 was the third 
consecutive year for a session of this type. Although the Presessions on 
Reading Research and Anthropological Field Methodology necessarily drew 
applicants with rather specific needs and interests (support for this is 
also found in the fact that about 40% of the persons applying to one or the 
other of these Presessions indicated no second choice) , they still received 
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4 p a sizeable number of applicants and were tied for third and fourth rank 

among the eleven Presessions. Because their content appeared to be either 
highly technical or specialized, Presessions # 4 , #9 and #7 probably appealed 
to fewer applicants than did the other Presessions', if it was not the case 
that these Presessions were technical or specialized, then we may have been 
unsuccessful in portraying them correctly in publicity. In the _ future ,_ i t 
might be wise to make detailed outlines of course c ontent available to. 
Dotential applicants upon request. One can not fully portray the nature of 
a Presession through the media of publicity used for the 1968 Presesstong 

Program . 
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Presessions Applications Tally 
(Frequencies of First and Second Choices) 
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Total Number of Applicants=743 
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DESCRIPTION OF THE APPLICANTS 



A random sample of 200 from a pool of 743 applicants was drawn 
for purposes of description of those researchers who apply to the AERA 
Presessions Program. (The characteristics of the participants in 
individual Presessions are described in the individual Presession 
reports to follow.^ 

Sex : Approximately 80% of the applicants to the Presessions Program 

are male. A random sample of 200 persons from the AERA Directory for 
1967-68 showed that 80.5% of the membership is male. Presumably, then, 
there is no "gender-bias" in the AERA Presessions Program. It might have 
been true that females find it less difficult to get away for five days to 
attend than males, for example. However, there is clearly no evidence for 
this . 



Age : In an accompanying figure, an age pyramid for males and females 

who applied to the Presessions Program is presented. 

Geographic Distribution : In an accompanying figure, the geographic 

distribution of the 743 applicants to the 1968 Presessions Program is 
depicted. As an example of how this figure is interpreted, note that 
8.5% of the 743 applicants came from the "West Coast" (Washington, Oregon, 
California, Nevada, and Arizona) or that 38.5% of the applicants came from 
the upper-Midwest (Minnesota, Wisconsin, Iowa, Missouri, Illinois, Indiana, 
Michigan, and Ohio) . 

The geographic distribution of applicants is out of proportion to 
their membership in AERA. The ratio of AERA members in the "upper-Midwest" 
to AERA members on the "West Coast" is two to one (approximately 1,800 for 
the former and 900 for the latter). However, there were almost five times 
as many Presessions applicants from the "upper-Midwest" as from the "West 
Coast". Approximately 26% (22.5% plus 3.5%) of the applicants came from 
the "East Coast" and "New England" . This region contributes almost the 
same number of members to AERA as does the region designated "upper- 
Midwest" in the figure. However, the "upper-Midwest" region (within 250 
miles of Chicago) contributed 1.6 applicants for every 1.0 applicants from 
the "East Coast" and "New England." These data support a "conve 
hypothesis" about attendance at a Presessions Program. It appears that 
travel distance (expense, time, etc.) are important factors in a potential 
applicant's decision to apply. An alternative explanation of the data is 
that people in the "upper-Midwest" are more likely to attend any AERA 
function- -regardless of location--since a history of Chicago-based Annual 
Meetings has instilled a "convention-going" habit in them. The 1969 Annual 
Meeting in Los Angeles will provide an opportunity to choose between these 
competing hypotheses. 

If the trend observed here is corroborated, AERA should consider 
whether the needs of its membership would be better served by one of the 
following plans: 

a. Conduct different Presessions at different locations around 



the country during the same week. 



b. Conduct the same Presession at different locations around 
the countrv on different occasions. 

c. Assist participants in defraying travel expenses to and from 
the Presessions Program. 

Place of Employment : Among the 743 applicants, the following 

distribution of places of employment were observed: 



Place of Employment Percent 



1. College or University 71% 

2. Public School Systems 15% 

3. Federal Government 6% 

4. State Departments of Education 4% 

5. Other (IBM, SRA, 3M, etc.) 4% 

100 % 



The bulk of the Presession applicants are employed by universities 
and colleges, as one might exp act. The second largest source of applicants 
was public school systems; it is encouraging that nearly one out of seven 
applicants came from and returned to public school systems. One applicant 
in ten came from either the federal or state government. The 4% of 
applicants labeled "other" came primarily from business and industry. The 
industry-based educational researcher appears to be a new phenomenon. The 
interest of these people, numbering approximately 30, should be encouraged 
in future AERA Presessions Programs. Specifically, the program should be 
publicized in the major industries and Presession proposals should be 
solicited from people working in the industrial setting. 

It appears, however, that the majority of Presession participants will 
continue to be drawn from the university-college setting. In one sense, 
this is highly desirable. As has been documented elsewhere in this report, 
there exists a substantial "spread cf effect" of the Presession instruction. 
Many participants return to universities to teach courses, advise graduate 
students, and consult with colleagues both inside and outside the academic 
setting. The opportunities for academic personnel to transmit skills and 
knowledge to many educational researchers are greater than for people in 
public school systems, state and federal government, and industry. 

Previous Attendance at Presessions : Approximately 15% of the applicants 

indicated that they had attended an AERA Presession in either 1966 or 1967. 

It is impossible to "interpret" this figure other than to say that it is 
"better" than 0% and probably not as high as it could be. 

Academic Training : About 81% of the applicants held an earned 

doctorate. Originally, participation in the Presessions Program required 
a doctorate, The intent of placing this requirement for attendance was 
to exclude full-time and part-time graduate students, and thus not encroach 
Upon the domain of higher education. However, it has become apparent that 
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t j although the requirement of the doctorate from participants was a way 

of avoiding an encroachment upon the graduate programs of colleges and 
universities, it excluded other worthy applicants. Many applicants 
(in public schools, laboratories, academic institutions, etc.) could 
make convincing cases that they needed the instruction offered in the 
Presessions Program and that to receive it would not usurp the role of 
a university graduate program. Most of these persons had no formal ties 
with such graduate programs and planned to have none. In a few instances, 
graduate students were admitted (with the blessing of their university) 
because their university could not offer the type of instruction offered in 
the Presessions Program (these instances were very few in number, however). 

The procedure adopted for the 1968 Presessions Program, namely to 
exclude "students enrolled in the graduate program of an institution of 
higher learning," seems best. Exceptions can be made when a student can 
show that the instruction offered in a Presession can not be duplicated 
on his campus. It would not be advisable in the future to require a 
doctorate of Presessions participants since this year one out of five 
applicants di d not have the degree . 

Research Productivity: On the average, the participants had published 

2.90 articles in scholarly ("refereed") journals and had directed 0.89 
funded research projects. The average holder of a doctorate in education 
probably publishes less than two articles in scholarly journals in his 
lifetime. The Presession participants have already shown signs of above 
average productivity early in their professional lives. Thus it would 
appear that the Presessions Program is primarily serving "researchers" 
who are refreshing or extending their research skills. In gene ral, it is 
not making "researchers" out of "non-researchers" . 

There appeared to be some slight evidence (not reported here) that che 
more "academically oriented" Presessions, e.g., #3, # 4, #5, and #11, 
attracted more productive researchers than the less academic (more applied 
and practical) Presessions, e.g., #2, #6, #7, #8, and #9. It would appear, 
then, that a program of Presessions could be offered that would appeal to 
those persons now producing little research (e.g., research on curriculum 
development, research on school finance, evaluation, etc.). 
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1 i Description c>f Participants of 1$68 AERA 

Presession on Research in Reading* 



* \ 



Characteristic 



Percentage 



Sex 


• 


Male 

39 




Female 
61 ’ 




- 


3936-1943 


19 / 4 / 4-1951 


1952-1959 


i 960 - 


v/ithout 


M.A. Degree 


2 


19 


39 


37 


3 


Ph.D. Degree 




4 


19 


54 


23 



Nature of 
Employment 



Professor 

4 



Assoc. Professor 

19 



Assist. Professor 

46 



Coord. , Res. 
Assist. , etc. 

28 



Percentage of 


0-50 51 - 75 


76 - 90 91 


O 

O 

1 


Time Allocated 






to Teaching 


26 19 


22 


33 


Courses 


Lang. Arts Research 


Develop . 


Remedia 1 


Taught 


9 5 ' 


35 


40 


Level of 
Courses 


Undergraduate 


Graduate 




Taught 


37 

♦ 


40 




Pr i M 3 r y 
Research 


Research Design Soc. 


Pedag. 


Psych. 


Interes t 


26 4 


56 


23 


Research Ar tides 








theses, or technic. 


0-2 3-5 


5-10 


10 + 


reports, pub. or 
unpub. 


33 28 


18 


20 


Funded Research 


0 


1 - 2 


3 + 


Projects 


59 


30 


7 
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O 

Description of Participants (cent.) 



Characteristic 




Percentage 




Profess iona 1 


IRA 


apa 


Ph.D.K 


Societies 


59 


1^ 


9 



* This does not include those who were accepted but did not attend. 
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INTRODUCTION 



Fifty-one educational researchers interested in doing research in 
reading instruction met for four and one-half days from 2 through 7 
February, 1968. Their object was to learn what research is being carried 
on in reading instruction, the methods and theoretical competencies necessary 
for the conduct of this research, arid the directions being taken by the 
leaders in this area. 

The sessions consisted of morning and afternoon lectures with the 
participants splitting into several seminars in the evenings. The lectures 
provided for organized presentations of the rationales and methods used in 
the various types of research discussed. The seminars were designed to permit 
the participants to explore in greater detail the topics in which they had 
special interests. There were totals of nine lectures and fourteen seminars. 

The participants exhibited a broad range on almost any dimension on which 
they were examined. The ranges were accounted for almost entirely by the fact 
that two fairly distinct populations are interested in research in reading. 

There was a substantial number of educational psychologists who exhibited their 
usual pattern of having a fair to good background in research design and learning 
and cognitive psychology but almost no knowledge of linguistics, verbal learning, 
instructional design, and reading curriculum. The second major group consisted 
of reading specialists who exhibited roughly the reverse pattern except that 
they also possessed too little knowledge of linguistics and verbal learning to 
permit them to understand the major independent variable of reading instruction, 
language. 



STAFF AND INSTRUCTION 



The instruction was planned around the characteristics anticipated 
(and actually observed) in the participants. Dr. Richard Venezky, who 
holds an M.A. in Psychology from Cornell and a Ph.D. in Linguistics from 
Stanford, introduced the participants to basic linguistic concepts, and 
di.scussed the methods and problems of studying both the grapho-phonemic 
relationships of the language and the learning of these relationships by 
children. Many of his illustrations came from his own research. 

Dr. E. B. Coleman, a well known psycholinguist who has been studying 
the problems of designing instruction in beginning reading, focused his 
attention on the relevance of both verbal learning and linguistic theory 
to the design of instruction. In his lectures and seminars he dealt with 
the problems of maximizing simultaneously the large numbers of response 
outcomes in beginning reading while minimizing undesirable outcomes and 
while dealing with irregularity in the spelling system. In his seminars he 
helped the participants design studies which will provide the scaling information 
necessary to design beginning reading instruction. 

Dr. Gene V. Glass, who has earned considerable respect for his work in 
evaluation and research design, concentrated his instruction on the design of 
experiments having both internal and external validity and on the selection 
of appropriate analyses. He also gave considerable attention to the logic 
underlying experimental analysis. 

Dr. John R. Bormuth dealt with three topics. First, he outlined the 
theoretical and methodological problems involved in attempts to rigorously 
represent and then analyze the cognitive processes involved in language 
comprehension. He presented some of the results of his work in developing 
rigorous descriptive devices. He also discussed theory and research methodology 
in readability and research in syntactic complexity. 

Dr. S. Jay Samuels discussed theory and research methodology in the 
experimental analysis of word recognition behaviors. He outlined the areas 
in which study should be conducted and illustrated, drawing on his own work, 
how this research could be carried out. 

Dr. Kenneth Goodman presented quite a different approach to the analysis 
of word recognition behaviors. In his work he attempts to construct a model 
of the word recognition process by correlating miscues the child makes while 
reading with the language in the text and the language habits of the reader. 

He presented his research methodology and discussed its advantages and limitations. 

The presession was held at the Abbey, a resort hotel on Lake Geneva in 
Wisconsin. This site was selected because it isolated the participants from 
the distractions of a large metropolitan area and threw the participants 
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together into a close association throughout the conference. At the same time 
it offered an indoor swimming pool and enough other recreational facilities to 
provide a modest amount of diversion for the participants. This facility 
achieved both of its intended objectives, and aside from a few problems in 
room reservations and room ventilation, it provided comfortable, convenient, 
and pleasant accommodations. 



4 V 




o 

ERIC 



33 



PARTICIPANTS 



Dr. Ira E. Aaron 
Professor of Education 
College of Education 
University of Georgia 
Athens, Georgia 

Dr. Wilbur S. Ames 
I3II Robert Ray Drive 
Columbia, Missouri 65201 

Dr. Rita Elizabeth Bergman 
Ashland College 

Dr. Mary T. Berry 

Middle Tennessee State University 

Sister Masseo M. Blesius 
Winona State College 

Dr. Emery P. Bliesmer 
Director, Clinical Reading 
Pennsylvania State University 
103 Educational and Psychology Center 
Building 11 

University Park, Pennsylvania 16802 

Dr. W. Archie Blount 
Winston-Salem State College 

Dr. Helen V. Bonnema 
Temple Buell College 

Dr. Beatrice J. Boose 
Virginia State College 
Norfolk Division 

Dr. Jean G. Boyce 
SUNY 

Dr. Mari J. K. Brown 
DePaul University 



Dr. Jane H. Cattersoh 
University of Saskatchewan 

Dr. Jean I. Caudle 
Professor of Education 
Wisconsin State University 
800 Algoma Blvd 
Oshkosh, Wisconsin 54901 

Sister Josephina Concannon 
Boston College 

Dr. Ruth C. Cook 
Mankato State College 

Dr. Robert K. Cornell 
Southwest Regional Laboratory 
Inglewood, California 90304 

Dr. Kay S. Earnhardt 
Emery University 
Atlanta Public Schools 

j Dr. Robert L. Emans 
Associate Professor of Curriculum 
and Instructions 
Temple University 
Philadelphia, Pennsylvania 19122 

Dr. Joseph L. Fearing 
University of Houston 

Sister Mary Julitta Fisch 
O.S.F. 

Dr. Margaret M. Fleming 
Supervisor, Bureau of 
Educational Research 
Cleveland Public Schools 
i 1380 E. 6th Street 
Cleveland, Ohio 44114 







Dr. A, Byron Callaway 
College of Education 
University of Georgia 
Athens, Georgia 30601 

Dr. Donald T. Cannon 
Creighton University 

Dr. Bartell W. Cardon 
Assistant Professor 
School Psychologist 
Penn State University 

119 EPC 11, University Park, Pa. 16802 



Dr. Flora Fowler 

East Tennessee State University 

Dr. Edith M. Gifford 
Assistant Professor of Education 
Oregon State University 
Corvallis, Oregon 97331 

Dr. William L. Goodwin 
Bucknell University 
Lewisburg, Pennsylvania 17837 
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Dr , Herbert M. Ha f frier 
Livonia Public Schools 

Dr. James W. Hall 
Northwestern University 

Dr. James R. Hengoed 
Assistant Professor 
School of Education 
Boston University 

Dr. Ruth Virginia Hintsalo 
Northern Michigan University 

Dr. Roderick A. Ironside 
Educational Testing Service 
Princeton, New Jersey 

Dr. Robert B. Kane 
Associate Professor 
Purdue University 
West Lafayette, Indiana 47907 

Dr. Ethel M. King 

Professor, University of Calgary 

Calgary Alberta, Canada 

Dr. Thelma M. Kloempken 
Winona State College 

Dr. David W, Knight 

University of Southern Mississippi 

Dr. Pose M. Lamb 

Purdue University 

West Lafayette, Indiana 47906 

Dr. Alma A. Leadbeater 
Abington School District 

Dr. Edith Levitt 
Columbia University 
New York, N.Y. 

Dr. Janet E. Lieberman 

Hunter College 

City University of New York 

Dr. Alvin J. Lowe 
Assistant Professor 
University of South Florida 
Tampa, Florida 33620 



Dr. Alan E. Maber 

Union Free School District 

Dr. George E. Mason 
Associate Professor 
University of Georgia 
Athens, Georgia 30601 

Dr. William J. Massey 
Homewood School and 
Bowie State College 

Dr. William B. McColly 
University of South 1 Carolina 

Dr. Edward P. Merryman 
Missouri University at St. Louis 

Dr. Wilma H. Miller 
Wisconsin State University 
La Crosse, Wisconsin 

Dr. Roland A. Montambeau 
Livonia Public Schools 

Dr. Lorraine L. Morgan 
Chatham College 

Dr. Shirley D. Myers 
Glynn County Schools 

Dr. Jeanne R. Nurss 
Emory University 

Dr. Gus P. Plessas 
Sacramento State College 

Dr. James A. Poteet 
Lincoln School 
814 North 14th Street 
Lafayette, Indiana 47904 

Dr. Joseph W. Quinn 
Coordinator 

Pupil Personnel Services 
Calgary Separate Schools 
Calgary, Alberta, Canada 

Dr. Wallace Z. Ramsey 

University of Missouri at St. Louis 

Dr. Ned H. Ratekin 
University of Northern Iowa 
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Dr. Jean E. Robertson 
University of Alberta 



Dr. Jane W. Torrey 
Connecticut College 
New London, Connecticut 



Sister Marie Colette Roy 

O.S.F. 

The Cardinal Stritch College 



Dr. Yancey L. Watkins 
Murray State University 



Dr. William D. Rutherford 
University of Chicago 
Chicago, Illinois 



Dr. Debrah M. Weiss 
Ontario Institute for Studies 



in Education 



Dr. David B. Ryuckman 
University of Michigan 



Dr. Joanna P. Williams 
University of Pennsylvania 
Philadelphia, Pennsylvania 



Dr. Adrian B. Sanford 

President, Educational Development Corporation 

200 California Avenue 

Palo Alto, California 94306 

Dr. Antusa P. Santos 
Mankato State College 

Dr. Bickley F. Simpson 
Lesley College 

Dr. Hazel D. Simpson 
Associate Professor 
College of Education 
Baldwin Hall 
University of Georgia 
Athens, Georgia 30601 

Dr. June J. Slobodian 
Livonia Schools 
Madonna College 
Eastern Michigan University 

Dr. Marjorie S. Snyder 
Child Study Center 
Kent State University 
Kent, Ohio 

Dr. Doris M. Stampe 
Ferguson-Florissant School District 
University of Missouri 

Dr. Laurel N. Tanner 
Hunter College 
University of New York 
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I. Materials mailed to participants 



A. 



1 



First mailing to participants, included: 

First letter to participants 

Description of Presession in Reading Instruction 
Background questionnaire 
Postcard questionnaire 
Brochure on The Abbey 



2 . 

3 . 

4 . 

5 . 



B. 

c. 



Follow-up on first letter to participants 



1 



2 . 



Second mailing to participants, included: 

Second letter to participants 
Tentative schedule 
Bus schedule 
.Bus preference 

Roommate preference . 

Roster of participants (see part 1 of materials; 



Materials handed out 

A. 



Schedule for lectures by Glass, Gene Glass, 1 p., schedule of Glass 
lectures (he was ill for final lecture) 



B. 



Notes on Sources of Internal and External Experimental Invalidity, 
Gene Glass, 6 pp., description of Campbell and Stanley's sources 
of internal invalidity and Bracht and Glass' sources of external 

inva 1 i di ty . 



C. 



Illustrations of Sources of Internal Invalidity, Gene Glass, 2 pp., 
examples for No. B. (above). 



D. 



The Dornan-Delacato Rationale: A Critical Analysis, Melvyn P. Robbins 

and Gene V. Glass, 43 pp., critique of Doman-Del cato ' s theory on the 
relationship between neurological organization and reading. 



E. 



A Critioue of Experiments on the Role of Neurological Organization 
in Reading Performance, Gene V. Glass and Melven P. Robbins, 3 PP* * 
critique of research designs of studies on Doman-Del acato theory. 



F. 



The External Validity of Comparative Experiments in Education and the 
Social Sciences, Glenn H. Bracht and Gene V. Glass, 40 pp., discusses 
sources of external invalidity in comparative experiments. 



G. 



The Experimental Unit and the Unit of Statistical Analysis: Compar- 

ative Experiments with Intact Groups, Gene V. Glass, S pp., discussion 
of appropriate experimental unit and unit of statistical analysis. 
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H. Comprehension Bibliography, John R. Bormuth, 7 pp., bibliography of 
sources on language comprehension. 

I. Summary of "On the Development of a Theory of Comprehension 
Instruction," John R. Bormuth, 2 pp., discussion of requirements 
for a theory of language comprehension, requirements for the task 
of teaching and measuring the language comprehension processes. 

J. Tables Accompanying "On the Development of a Theory of Comprehension 
Ins tr uct-ion,!' John R. Bormuth, 20 pp. , tables for No. I. 

K. Collecting a Data Base for an Educational Technology, //2, Rank-order ing 
Word Classes According to Response Availability, E. B. Coleman and 

T. C. Potter, 12 pp., discusses rank-ordering word classes accordint to 
their response availability. 



L.* Effects of Different Kinds of Visual Discrimination Training on 

Learning to Read Words, Ethel M. King, 9 pp.j Compared kindergarten 
words learning to read 4 words folio-wing different kinds of visual, 
discrimination. 



M. Variables in Early Discrimination Learning: I. Motor Responses in 

the Training of A Left-Right Discrimination, Wendell E. Jeffrey, 

7 pp., studies the transfer effect from learning a discriminative 
motor response to the more difficult labeling task. 

N. The Orienting Reflex and Attention in Cognitive Development, Wendell 
E. Jeffrey, 17 pp., Discussion of previous research. 

O. Writing as Pretraining for Association Learning, Harry Levin, John 
S. V/atson, and Margaret Feldman, 4 pp. , tested effects of writing as 
pretraining for association learning. 

P. Recent Research in Visual Discrimination: Implications for Beginning 

Reading, Siegmar Muehi and Ethel M. King, 15 pp.j discusses kinds of 
training that arc most effective in helping children learn to discrim- 
ote among graphic patterns. 

Q. The Effects of Visual Discrimination Pretraining on Learning to Read 
A Vocabulary List in Kindergarten Children, Siegmar Muehi, 5 pp.j 
studied the types of sensory experiences which facilitate visual 
discrimination among printed words occurring in a reading task. 

R. The effects of Letter-name Knowledge on Learning to Read a Word List 
in Kindergarten Children, Siegmar Muehi, 6 pp. , studied the effects 
of learning letter names on the subsequent acquisition of word name 
associations. 




S. The Effect of Attention and Motor Response Pretraining on Learning 

to Discriminate B arid D in Kindergarten Children, Lois N. Hendrickson 
and Siegmar Muehi, 6 pp., compared 3 treatment groups of kindergarten 
children learning names for the letters "b" and "d". 

* L through S were mailed. to participants, to be read before the prescssion. 
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T. The Effect of Word Associations on the Recognition of Flashed Words, 
S. Jay Samuels, 3 pp*, tables to accompany lecture. 
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RESULTS OF STAFF QUESTIONNAIRE 

AERA 1968 Research Training Presessions Program 

Presession Critique for Staff Members 
(For Directors, Instructors and Assistants) 



Percent 



CD 

H 
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o 

4J 

o 

CO 

•H 

■U 
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CO 

•rl 

4J 

CO 

CO 

Cl 



Indicate your observation and judgment by checking each 
item in one column at the left. Items not applicable or 
not subject to your observation should be omitted. Be 
frank. 



Environmental conditions 



33 


67 


0 


0 


a. 


Classroom spaces 


40 


60 


0 


0 


b. 


Work spaces 


67 


33 


0 


0 


c. 


Living quarters 


33 


67 


0 


0 


d. 


Teaching equipment, aids (chalk boards, publ 












system, etc.) 


20 


60 


0 


20 


e. 


Resource material, library 


33 


33 


33 


0 


f. 


Eating facilities 








2. 


Participants 


33 


67 


0 


0 


a. 


Appropriateness of academic backgrounds 


0 


67 


33 


0 


b. 


Sufficiency of research experience 


83 


17 


0 


0 


c. 


Willingness to work 


83 


17 


0 


0 


d. 


Intellectual curiosity 


67 


33 


0 


0 


e. 


Concern for applicability of techniques 


40 


20 


0 


40 


f. 


Aspiration 


0 


60 


20 


20 


g- 


Immediate preparation for Presession 








3. 


Organization 


83 


17 


0 


0 


a . 


Adequacy of notice to prospective applicants 


100 


0 


0 


0 


b. 


Sufficiency of preplanning 


83 


17 


0 


0 


c. 


Smoothness of operation 


67 


17 


17 


0 


d. 


Adaptability to obstacles and feedback 


60 


40 


0 


0 


e. 


Sensitivity to grievances 


60 


20 


20 


0 


f. 


Adequacy of financial support 








4 . 


Schedule 


40 


60 


0 


0 


a . 


Appropriateness of five days for the job 


83 


17 


0 


0 


b. 


Time spent efficiently 


50 


50 


0 


0 


c. 


Events sequenced appropriately 


67 


33 


0 


0 


d. 


Punctuality 


67 


33 


0 


0 


e. 


Balance between formal, informal affairs 


80 


20 


0 


0 


f. 


Quantity of discussions 


40 


40 


20 


0 


g- 


Quality of discussions 


60 


40 


0 


0 


h. 


Quality of formal presentations 


50 


50 


0 


0 


i. 


Unobtrusiveness of evaluation efforts 


40 


60 


0 


0 


j* 


Methods of evaluation 
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5. 


Outcomes 


80 


20 


0 


0 


a. Intended content was actually taught 


100 


0 


0 


0 


b. Increase in participant understanding 


60 


40 


0 


0 


c. Improvement in attitude toward research 


80 


20 


0 


0 


d. Personal associations initiated 

1 Many people indicated plan to continue 



6. In general was the Presession well organized? 

Yes - 100% No - 0% 

7. Were the facilities suitable for the activities which you 
had planned? If not, specify. 

Yes - 100% No - 0% 

8. Should Presessions be limited to the same hotel, or the 
same city, in which the annual meetings will be held? 

Yes - 0% No - 100% 

9. Were you to do the same assignment over, in what major ways, if 
any, would you change your contribution? 

1-Not predictable at the time 
1-More selective in admitting applicants 
1-Cover fewer research studies 
1-Concentrate more on research studies 




10. Do you wish that the Director had made firmer arrangements to 
assure participants and you of the staff opportunity to meet 
in pairs or small groups? 

Yes - 0% No - 100% 

11. Were the objectives you set for yourself during the Presession 
attaine d? 

Yes - 83% No -17% 

12. Are you inclined to urge your colleagues to become staff members 
for such an institute or Presession? 

Yes - 100% No - 0% 

13 In what ways, if any, did you as a staff member benefit personally 
as a result of your participation in this Presession? 

2-1 learned a great deal from both the other instructors and the 
participants . 

1-1 enjoyed the opportunity to meet more of the people who are 
doing research in reading. 

1-1 found new direction for my research. 

1-1 met colleagues with interest in same topics - both professional 
and personal relationships. 

1- Sounding board, potential cooperative researcn contacts. 

1-This staff participation was seen as a prestigious activity- 
and a little status makes people regard ideas with respect. 
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I gained a better appreciation than I had previously for the 
educational researcher -- reading specialist and her (his) 
problems and limitations. 

1-1 met many interesting people. The Presession offered me another 
opportunity to look at my own work in preparation for presenting 
it to the group. 

It wasn't to my benefit to be away from campus so long at the 
beginning of second semester-- I'm still trying to catch up. 
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Percentage of 
Responses 
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ACRA 1968 Research Training Presessions Program 
Participant Evaluation Form 



Indicate your observation and judgment by checking each 
item in one of the indicated (underlined) columns at 
the left. Do not make a mark in a column v/here an X 
appears. Your frank and honest evaluation can only 
benefit everyone concerned. Do not identify yourself 
by name unless you prefer to do so. 
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a. 
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75 


21 


83 


12 


25 


75 


54 


42 


79 


17 


79 


17 


75 


8 


58 


33 


39 


58 


67 


25 


71 


25 


83 


8. 


79 
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Environment and Faci 1 i t ies 

i i ** i i ■ — m. ■ m* 

How satisfactory was the relative availability of 
books and journals in promoting yQur attempts to 
master the content of this session? 

How satisfactory were the reproduced materials given 
to you by the staff in promoting your attempts to 
master the content? 



either alone or in small groups? 

b* V/as your noom satisfactory? 

• 

Mark v;i th a p1us(+) those features of the meeting 
rooms and facilities of The Abbey which facilitated 
learning and with a minus(-) those features- which 
•were inadequate or not conducive to learning: 

Time schedule 

___ Classroom spaces 

__ _ Venti lation in classrooms 

___ Temperature of classrooms 

__ Acoustics in classrooms for hearing lecturers^ 

___ Acoustics in classrooms during discussions 
C I ass room size 

(if negative) too iarge too small 

_ Space to spread out materials during sessions 
___ Chalkboards 

^Electronic teaching aids, 

___ Abi 1 i ty to see lecturers 

Lack of competing attractions (as in a city) 

____ Attraction of Abbey's recreational features 

b# Additional features (indicate whether positive or 
negative): 



(Abbey's recreational features facilitated learning since they ware quite unsatisfac toi 
(Room ventilation poor -- hard to control temperature) 
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(Schedule too tight to relax at all.) 

(Half-day free about Mon. p.m. v/ould give time to take advantage of recreation in area 
c.g. U. of Chicago observatory.) 



cn <u 
c 4-* 
O ru 
•— D 
cr 
<i) 

•u 
< 



•M 

I- 

o 

JC 

</) 



o 

o 



V 

L 

o 



0 ) 

E 

10 

to 



JL $4. 46 



0 

.0 


V) 

a> 


0 




1- 


>- 






± 


XX 


XX 


4. 


li 


XX 


XX 




JD 


XX 


XX 


m i 

5. 


XX 


21 


21 


(?) 

4 


XX 


21 


11 


6. 


XX 


21 


1 7 


7. 


XX 


21 


11 




XX 


88 


JB 




XX 


'll 


21 


8. 


XX 


88 


11 


9. 


t n 
i n 
V 


t/> 

<u 


O 






>- 






46. 


XX 


XX 


10. 


XX 


46. 


4_6_ 


(?) 

8 ’ 


J5 


XX 


XX 


8 



Schedu l inq and Organisatio n 

a. Appropriateness of five days as the length of time 
to leave your work at home for the purpose of 
attending^ this session. 

b. Appropriateness of five days as the length of time 
to learn much of the content of this session. 



lectures. 

b. Were the lectures scheduled in an appropriate sequence? 

/ 1 1 \ * l 

l what sequence) * ' 

Did you have sufficient opportunities to interact with 
other participants? 



b. V/ere the instructors sufficiently accessible and 
approachable for you to get the individual attention 
that you desired? 

c. Was it helpful to have graduate student assistants 
present? (The were magnifient.) 



during the session (and at ’this moment) conducted so as 
•to avoid interference with your work here? 



(Very) 



b. 

c. 



Amount of time which you had to pursue activities of 
your own choosing. • 

ft 

Did you like meeting in the evening after dinner? 

Should the number of meetings per day have been 
different? 
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Content and Pr esentat ion 

Did the content of the lectures and readings in each 
of the following areas presuppose previous training 
different than that which you had? 

Previous reading research studies 

Word rccogni ti on 

Comprehension 

L inguist ics 

Research design (at too low a level) 

Evaluation of instructional programs 

This amount of training in these area^ i shoul d have been 
presupposed. * * 

(One-sided presentation of language as r, out there" -- 
Goodman helped to counterbalance.) 

(Many didn't understand G lass 1 presentation.) 
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Do 



To what extent was the content of the lectures and readings 
relevant to what you hoped to accomplish during the session? 

Lectures were stimulating and interesting (Goodman, Samuels 

. . ' Venezky) 

Lecturers were competent to speak on the subjects 

assigned to them. 

Lecturers were well prepared (Goodman good) 

Seminars were successful (Some) 

Discussions were successful (Some) 

V/ere you satisfied with’tfve group of participants? 

List the reasons for any disapointrr.ent, if you wish! 

(see attached sheet^ 



days to make it more likely that you will leave your 
present position of employment? 
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J* 


XX 


XX 


20. 


How successful do you think the Presession 
in accompl ishing the staff's objectives? 


has 


been 


8 


A? 


j*6 


Jl 


XX 


XX 


2 1 . 


Do you think your understanding of reading theory and 
research has been enriched in these five days? 


8 


XX 


XX 


XX 


05 


17 


22. 


* 

Do you plan to conduct a research study in 
of reading soon? 


the 


field 


ft 


XX 


XX 


XX 


£7 


17 


23. 


V/as the study you plan to conduct influenced by what 
you learned this’ week? N \ \ ,V' 




XX 


XX 


XX 


CO I 


£ 




In design? 








XX 


XX 


XX 


£ 


29 




In objectives? 








XX 


XX 


XX 


£ 


21 




In basic concept? 
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All comments added by participants are in parentheses. 




« 



More responses to 3* b. : 



(Committment re room facilities — single room not available as 

two days.) •. 

(Rooms were fine for a vacation, but need a desk and chairs for 

(Acoustics in rooms not conducive to study when adjoining rooms 

(No soap in room had to ask for soap.) 



promised for first 
work.) 

contain teen-agers.) 

Vf 



(Could have had more competent research in fields discussed by Coleman, Samuels.) 



Comments on 8: ' [ic" 

(Would like feedback.) - „ , . . . > 

(I doubt that the exams are getting at achievement of primary objectives.; 

(Too detailed many of the responses reflected previous knowledge far more than 

knowledge attained here.) # ’ 

(Evaluation not really necessary or appropriate.) 



Comments on 14, b.: 
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SUMMARY AND RECOMMENDATIONS 



The presession was well received by the participants. Almost one-fourth 
of the participants have written letters to the director expressing their 
satisfaction with the content of the presession and with the other arrangements. 
More than twice that number expressed similar sentiments directly to the director 
before leaving. Tabulations from the participants' questionnaires (which were 
filled out anonymously) seem to substantiate these observations. 

I, the director, personally was pleased with the instructors and the 
participants. In retrospect, I do not believe I would design the same kind of 
presession if I were to set out again to train persons to do research in reading. 
I question whether it is possible in a five day period to accomplish such an 
objective. In order to perform high quality ard significant research in reading 
instruction the researcher must have a good substantive background 
in linguistics, psychology, verbal learning, psychology of learning and 
cognition, and reading curriculum and instruction as well as a knowledge of 
research design. It might be realistic to think that one of these topics could 
be covered at an adequate level in five days. It might even be possible to 
instruct students in how that area of study is relevant to research in reading 
instruction, but to attempt to do more than that seems, after the fact, futile. 



Given the same task over again, I would select just one of these substantive 
areas, say linguistics, give the students enough instruction in it to provide 
them with both a basic understanding of the discipline and a good idea of what 
further study they had to do, then I would devote the remainder of the session 
to showing how the theory and research methodology of that area can be used to 
perform research relevant to reading instruction. 

In summary, the participants seemed well satisfied with the presession. 

But I believe their satisfaction stemmed from the rather heady stimulation one 
gets from encountering new and powerful approaches to some old and difficult 
problems, and I do not believe the participants were able to absorb enough 
organized knowledge in the brief amount of time available to enable them to 
materially improve their research. The instructors were well satisfied with 
their performances and for good reason -- it was uniformly excellent. But the 
director was quite dissatisfied with his design of the presession. He now believes 
that the topic of reading instruction is entirely too broad to permit work in 
sufficient depth to make any real difference in the participants' research 
behaviors. He recommends that, if there are any future presessions in the area 
of reading, the director should consider limiting the scope to a single research 
discipline and restrict admission to the presession to those who are otherwise 
competent to do research. This should be done even if it means eliminating all 
but 20 or 25 applictn^s. 
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INTRODUCTION 



The purpose of this report is to present an evaluation of the activities 
carried out in connection with the 1968 AERA Presession on Educational Research 
Management procedures. The report is directed primarily to the Chairman of 
the 1968 AERA Presession Committee, Dr. Gene Glass of the University of 
Colorado, for his use in preparing a final report on all Presessions. This 
report is also directed to the participants in the Presessicn so that they might 
see the results of the evaluative scales that they completed during the final 
session of the total program. 

Some of the evaluative materials have been summarized rather tnan presented 
in detailed form since the request made by the Presession Chairman focused 
primarily on such summaries. Certain other items have been included in the 
report, such as list of handout materials of which the participants already 
have knowledge, on the basis of the request from the Chairman of the AERA 
P'esession Committee. 

The report consists of four separate sections subsequent to this Introductory 
Section. The second section presents an outline of the Presession and includes 
items relating to the schedule, advanced materials, handout materials, partici- 
pant characteristics, and a list of participants. The third section consists 
of summary r'oorts of participant evaluation of the Presession. Included is 
a summary of the AERA Evaluation Sheet, the Institute Evaluation form, and 
an evaluation of the management game developed by the Educational Program 
Management Center and used in connection with the Presession. The fourth section 
consists of an evaluation of the Presession on the part of the staff. This 
section consists of a summary of the individual staff reports prepared by the 
four staff members presenting instruction of the Presession along with summary 
remarks and recommendations by the AERA Presession Director. The final section 
of the report presents a summary of the total program. 
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PRESESSION OUTLINE 



The Presession staff consists of four staff members. Two of the 
staff were full-time faculty of the members of the College of Education, 

The Ohio State University; the other two staff members were graduate 
students serving as Research Associates in the Educational Research 
Management Center, The College of Education,, The Ohio State University. 

The participating staff members were as follows: 

Dr. Desmond L. Cook - Director 
Dr. Edwin Novak - Staff 
Mr. Duane Dillman - Staff 
Mr. William Loeber - Staff 

The Presession schedule presented in Table 1 represents how the 
program was planned and is also a good representation of what was actually 
completed. The organization of this schedule reflected four general areas 
of instruction. The materials presented on Saturday were intended to 
provide a framework for the remaining material and provide a common 
orientation to the participants. The presentations schedule for Sunday 
and Monday focused on technique for planning a project and Tuesday's 
presentations focused on control techniques. The presentations on Wednesday 
were intended to tie together the previous concepts and also to present an 
introduction to the management of several projects. All material was 
presented with the exception of the first unit on Wednesday which was 
cancelled due to an unexpected illness of one of the Presession staff. 

Slight deviations from a specified time table occurred due to the unexpected 
complexity of certaim simulation exercise materials but such deviations were 
at a minimum. 

The advance literature was distributed with two objectives in mind. 

The papers by Cook (1) and Frederickson (3) were intended as supplementary 
materials to the lectures presented on the first day. The monograph by 
Cook (2) was the most important of the advance materials in that reading it 
would give the participant a better understanding of the management techniques 
presented in the second, third, and fourth days of the session, A biblio- 
graphical list of materials mailed out is presented below. 

1. Cook, Desmond L., "The Use of Systems Analysis and Management 
Techniques in Program Planning and Evaluation", Symposium, 
Chapman College, Orange, California, 1967. 

2. Cook, Desmond L., Program Evalua tion and Review Technique 
Applications in Education, U. S. Government Printing 
Office, Cooperative Research Monograph Number 17, 1966. 

3. Frederickson, Norman 0. , "The Administration of an Educational 
Research Program", 1966. Reprint from AERA Publication. 
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The materials listed below were distributed during the Presession. 

They were intended to provide additional supporting information to 
lecture materials and also to provide alternative approaches to some of 
the concepts presented. 

1. Cook, Desmond L., "Better Project Planning and Control Through 

the Use of Systems Analysis and Management Techniques", 

USOE Symposium, Washington, D. C., 1967. 

2. Katzenbach, Jr., Edward L., "Program Budgeting for Sponsored 

Research", Paper presented at Joint Conference sponsored 
by American Council on Education and Johns Hopkins University 
in 1965. 

3. Kaufman, Roger A. and Robert E. Corrigan, "What is the System 

Approach and What's In It for Administrators?", Symposium, 
Chapman College, Orange, California, 1967. 

4. Meals, Donald W. , "Heuristic Models for Systems Planning", 

Phi Delta Kappan, January, 1967. 

5. The A, B, C's of PPBS. Reproduced from The Secretary's Letter , 

Volume 1, Number 3, Department of Health, Education, and 
Welfare, July, 1967. 

Data regarding the participants by location or setting of their work, 
an:, major responsibilities in their work are presented in Table 2. Since 
several participants had more than one major responsibility and were 
possibly associated with more than one location, the frequency totals 
exceeded the total number of participants of the total group participants. 
Thirty-nine participants possessed the doctorate degree and 6 the master 
degree. There were forty- three men and two women at the presession. 

A list of participants' names and addresses is presented as Appendix A 
to this report. 



PARTICIPANT EVALUATION OF THE PRESESSION 

A decision was made early by the staff in planning for the resession 
that any evaluation procedures centered around measurement of student 
achievement were basically inappropriate to the nature and purpose of the 
presession. It was held that such sessions were devoted primarily to "up- 
dating" professional persons and that measurement efforts directed towards 
determining how much growth had taken place would work against the atmos- 
phere desired by the Presession staff. As a consequence, no measures or 
instruments were developed which would assess growth. The evaluation 
procedures developed focused primarily upon securing feedback regarding 
the Presession itself and selected instructional materials. This section 
presents the results of these evaluations by the participants. 
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Three of the participants were located both in a university and State Department of Education. 
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The participant evaluation system for this presession consisted of 
three forms filled out by the participants at the conclusion of the 
presession. The three forms were treated as unrelated devices and no 
attempt was made to interrelate the results and formulate any broad 
generalizations. However, generalizations were drawn from the results 
of each separate evaluation form. 

The Participant Evaluation Form was developed by AERA Presession 
Committee and was a standard form used by all presessions. It thus serves 
as a common basis for judging the effectivenessness of each Presession 
against all the others. 

The Institute Evaluation Form, also used in the 1967 Presession, is 
less specific than the AERA Participant Evaluation and did not seek specific 
suggestions for improvement of the Presession. The participants were thus 
freer to criticize the Presession since they were under no obligation to 
offer alternative plans. 

The third evaluation form used was the Management Game Evaluation. 

The Management Game was introduced in lieu of programmed exercises. It 
was hoped the game would not only encourage greater involvement but also 
allow the participants more flexibility in using their individual skills 
in solving management problems. The evaluation form was intended to elicit 
the participants' reactions to the strong and weak points of the game as 
a learning device. 

A. AERA Evaluation Form 

Over four-fifths of all the participants had high praise for the 
Presession in general. Following is a breakdown of specific facets of 
the Presession and the evaluation of these aspects by the participants. 



1. Facilities . All but a few of the participants had some favorable 
comments on the facilities, but over three-fourths also mentioned that 
the rooms were too hot and stuffy most of the time. Features which were 
most frequently mentioned as conducive to learning were adequate size of 
the meeting rooms and the presence of round tables at the Session. Several 
participants mentioned that meeting rooms for the small groups should have 
all had a blackboard. 

2. Sched uling and Length of Presession . The number and time of 
meetings was rated as excellent by over three- fourths of the participants, 
however several objected to the length of some of the meetings. Several 
participants also felt that the lectures (90 minutes) were too long. 

Over three- fourths , however, praised the competence and preparedness of 

the lecturers. 

The participants were split on their evaluation of the length of the 
Presession itself (five days). Half felt that the Presession should have 
been condensed into a 3-4 day period, while the other half felt that five 
days was not too long to leave their work at home and attend the Presession 
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It is interesting to note that although half of the participants would have 
preferred a shorter session, over 80 per cent stated that five days was not too 
long a period in which to learn much of the content of the Session. Several 
participants suggested mailing more content materials in advance and condensing 
the general information given during the first day. 

3. A dvance Materials and Content . All but one of the participants agreed 
that the advance materials that were sent out, especially the monograph and 
lecture handouts were quite helpful. Several participants suggested that many 
of the visuals be duplicated as handouts, thus cutting down on tedious note- 
taking. 

Ove;r four-fifths of the participants felt that the readings and lectures 
were relevant to what they hoped to accomplish and did not presuppose far more 
previous training than they had. 

4. Interacti on. Over three-fourths of the participants also praised 

the opportunities for interacting with other participants during the Presession. 

However, several participants stated that most of the interaction was within 

the same group with little inter-group interaction. About one-third indicated 

they will possibly collaborate on research with someone they met at the Presession; over 

over 90 per cent said they would maintain soitK sort of contact with at least 

one of the Presession staff. 

5. Reaction to Staff . All the participants with one exception, also 

had high praise for the cooperation and help from the staff. Several participants 
suggested that more direct attention by the staff, to the small groups would 
strengthen the Presession even more. Over half also indicated that graduate 
assistants were helpful a*t the Presession but about one- fourth suggested that 
graduate assistants not be used as instructors. 

B. Institute Evaluation 

General participant response to the Institute was highly favorable. In 
summarizing the responses of the participants, a weighted scale was used. All 
questions were interpreted as being either favorable or unfavorable to the 
Institute. Five points were awarded to each strongly favorable response, 4 
points to a favorable response and so forth with one point for strongly unfavorable 
responses. Using this system, responses to each question were totaled and then 
averaged. Table 3 presents a summary of participant responses to each statement. 

A comparison of 1967 and 1968 Presession responses on this same instrument 
is shown in Table 4. A similar response pattern for each statement can be 
observed. However, 1968 scores were not as high as 1967 scores. Wide differences 
in program content make direct comparisons between the two sessions somewhat 
difficult but the marked similarity of response patterns does raise a question 
as to the value and usefulness of the instrument. 

C. Management Game Evaluation 

All but a few of the participants rated the administration, organization, 
and realism of the management game quite highly. Some participants suggested, 
however, that a general introduction as to how the games operate would be helpful. 
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Table 4. Comparison of 1968 and 196/ Presession 
Institute Evaluation Form Responses 



1967 


1 968 


Statement 


Scale Vdlue 


Statement 


Sea 1 6 Val ua 


1 


4.54 


1 


3.98 


2 


4.31 


2 


3.41 


3 


4.06 


3 


3.22 


4 


4.1 0 


4 


3.44 


5 


3.90 


5 


3.78 


6 


4.67 


6 


4.68 


7 


4.56 


7 


4.34 


8 


4.02 


8 


3.85 


9 


3.60 


9 


3.02 •' 


10 


3.96 


10 


3.71 


11 


4.58 


11 


3.54 


12 


4.48 


12 


3.90 


13 


3.88 


13 


3.85 


14 


3.25 


14 


3.68 


15 


4.35 


15 


3.68 


16 


4.21 


16 


3.68 


17 


3.85 


17 


3.54 


18 


3.58 


18 


3.80 


19 


3 . 98 


19 


4.05 


20 


3.65 


20 


3.73 


21 


4.42 


i 21 


3.80 


22 


4.10 


22 


3.56 


23 


4.31 


23 


4.10 


24 


3.83 


24 


3.44 


25 


4.50 


25 


4.10 


26 


4.27 


! 26 


4.15 


27 


4.77 


27 


4.41 


28 


4.71 


28 


4.56 
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Over half the participants requested either a less complex problem for the 
game or an introductory "canned* exercise. 

There was wide disagreement among the participants concerning the time 
allotted for each session of the game. About three- fourths felt that the 
sessions were either too long (40%) or too short (60%) . Several pointed out 
that the program definition and networking session and the budgeting session 
should be extended. It was also suggested that the problem and introductory 
material be distributed the night before for study. Several participants 
complained that the lecturing was too elementary and the sessions tended to 
drag toward the end unless the participants were all forced to think. 

Over half the participants felt the information in the organization 
description and action memora-idums was both clear and helpful, although a 
few suggested that it be more specific. 

Almost all the participants indicated a low explication of the roles to 
be played but a good correlation between the game and the preceding instructional 
sessions. Several participants suggested, however, that the staff furnish more 
specific definitive information about the roles. It was felt that the staff 
did not fully provide for role-playing within each group. 

The participants indicated a generally negative evaluation regarding the 
amount of feedback provided at the end of the game. Several suggested either 
allowing more time or increasing the size of the staff so that feedback could 
be given after each stage in the game. It was also suggested that a staff 
member or experience! participant be assigned to each group. It was felt that 

the lack of feedback as to how they were doing was the weakest point in the 
game . 



Over half the participants thought the end products of each session were 
clear, but many pointed out that any lack of clarity was due to the complexity 
of the problem and lack of adequate time, thus keeping some groups from realizing 
their objectives. 

A general summary of the evaluation would indicate a favorable response 
from the participants, with the exception of the comments noted above concerning 
the complexity of the problem, time factor, and amount of feedback from the staff. 

SUMMARY OF STAFF REPORTS 

btaff responses on the AERA Presession Critique for Staff Members were all 
satisfactory or commendable except f or one unsatisfactory response core erning 
eating facilities. The staff felt that the objectives of the Presession were 
accomplished and that the participants gained an understanding of the material 
presented. A summary of staff critique is presented as Appendix B. 

It was realized that the management game developed for this Presession needs 
revision. The material used, although taken from an actual situation, led to 
initial confusion and delay with regard to developing the required plan. This 
was not intended or anticipated and as a result subsequent portions of the 
game had to be kept less detailed than originally planned. Revision of the 
game is now in process. 
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Reorganization of general program content is not planned. However, the 
Presession did point out needs for minor revisions of lesson materials, visuals, 
and handout materials. 



DIRECTOR 1 S REPORT 



1- Preliminaries . Having conducted Presessions in both 1967 and 1968, 
it is felt that the early organization and finalization of Presession planning 
by Gene Glass was most beneficial to the preparation for the 1968 Presession. 

The early selection of Presession proposals and subsequent funding authori- 
zations made it possible to select a staff well in advance and allowed adequate 
time for the development and organization of the presessions. Early site 
selection eliminated last minute problems in this area. The Presession 
Directors' meeting held in September was particularly helpful with regard to 
the above mentioned points as well as giving all Directors an opportunity 
to meet and agree on most administrative details. 

The early organization of all Presessions also aided the local preparations. 
Presession staff members were able to be selected from the staff of the Educational 
Research Management Center at the Ohio State University which greatly facilitated 
coordination of the program development. In addition, the fact that four local 
staff members were available by means of the budget was of great value in planning 
and coordinating the instructional sequence and content. 

2* Actual Presession . In general, the meeting room and living quarters 
were quite satisfactory. Of particular value was the large size of the room 
and the round table set-up which was well suited to small group work. The 
following problems were encountered. 

(a) Room temperature was generally too high at the beginning of 
each day. It often took one or two hours before the hotel 
engineer could be obtained and a change made. 

(b) The meeting room was used by another group during at least one 
evening, requiring collection and storage of teaching materials 
and equipment set up around the room. 

(c) The room was poorly set up for use of an overhead and movie 
projector. A large portable screen was provided which was too 
low to be seen by all participants. Speakers were obliged to 
move back and forth from the podium in order to place visuals 
on the overhead projector. Speakers were also obliged to move 
away from the podium when talking cxtemperaneously and using the 
overhead. A portable microphone was of great value in these 
instances . 

The presentations went as scheduled with a few minor exceptions. Progress 
in the management game exercise was slower than expected and certain portions 
had to be kept more general than planned. In addition, one of the senior 
staff members became ill on the last day and his final presentation was omitted. 
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The material mailed to participants in advance was meant to be read before 
attending the Presession. It was obvious by many of the questions asked that 
this had not been done by a fair number of participants. As a result, the 
staff is considering administering a pre-test at future sessions as a check 
on the value of advanced materials. 

3. Recommendations . On the basis of both the 1967 and the 1968 Presessions, 
it is recommended that the Presession length should be the Director's pre- 
rogative as opposed to the present five-day module. Several participants 
favored a shorter but tighter schedule. 

The application blank for the Presession tends to be too much oriented 
toward participants' possession of an experimental research background. This 
should be de-emphasized in favor of more general information which would be 
more pertinent to the wide variety of sessions conducted. 

It would be helpful to know participant objectives for and expectations 
from attendance prior to their selection. This would enhance the selection 
process and possibly the program structure. 

A general social hour at either the start and/or finish of all the Presessions 
is recommended to give participants from the several Presessions an opportunity 
to become acquainted . 



SUMMARY 



The composite reaction of participants and staff of the Presession was 
quite favorable. Suggestions for improvement were generally helpful and 
have been taken into consideration for future programs. Administrative 
procedures established by Gene Glass facilitated early planning and organ- 
ization of the individual sessions. A list of major comments, recommendations, 
and conclusions appears below: 

A. Presession Outline 

1. The instructional schedule was followed with only minor deviations 
and all material was presented with one exception. 

2. Advance materials did not appear to have been read by all partic- 
ipants and the staff is considering a pre-test for future 
sessions. 

3. Participant job titles were quite varied and their locations 
represented a good cross-section of educational institutions. 

B. Participant Evaluation 

1. Positive comments centered on the practical value of the 
material presented. 

2. Negative comments focused on the excessive heat in the 
meeting room and excessive length of certain lectures. 
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3. Comments on the management game centered on its difficulty and 
complexity for the time allowed. 

C. Staff Evaluation 

1. All staff members felt the objectives of the Presession were 
accomplished. 

2. In comparing the 1968 Presession with the 1967 Presession, the 
Director felt that general organization was greatly improved. 

3. The application blank for future PresesSions should emphasize more 
general applicant background information and request a statement of 
applicant objectives for attendance. 
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APPENDIX A 



1968 AERA Presession 
Participant List 



Dr. Roy D. Acker 
Director of Research 
Eastern Kentucky University 
Richmond, Kentucky 40475 

Dr. William H. Ashbaugh 
Psychological Services and 
Educational Research 
Milwaukee Public Schools 
Milwaukee, Wisconsin 53208 

Dr. Gilbert R. Austin, Director 
Bureau of Educational Research 
and Testing Service 
University of New Hampshire 
Box Q. 

Durham, New Hampshire 03824 

Dr. A. Edward Blackhurst 
Special Education Instructional 
Materials Center 
University of Kentucky 
641 South Limestone Street 
Lexington, Kentucky 40506 

Dr. George L. Brandon, Head 
Vocational Education Department 
250 Chambers Building 
Pennsylvania State University 
University Park, Pennsylvania 1 6802 

Mrs. Marjorie B. Brodt, Director 
Experimental Model School Project 
200 Tranquil Avenue 
Charlotte, North Carolina 28209 

Dr. Roderick E. Chisholm, Director 
Special Programs 
St . Mary's Col 1 ege 
Winona, Minnesota 55987 



Coff i ng 
School of 



Mr. Richard T. 

Assistant Dean, 

University of Massachusetts 
Amherst, Massachusetts 01002 



Educat i on 



Dr. John M. Coulson, Assoc. Director 
Regional Planning Service 
443 S. Gulph Road 

King of Prussia, Pennsylvania 19406 

Sister Crispin, Principal 
Visitation High School 
900 W. Garfield 
Chicago, Illinois 60609 

Dr. Floyd G. Delon 
Associate Director of South 

Central Regional Educational Lab. 

302 National Old Line Building 
Little Rock, Arkansas 72201 

Dr. Neil S. Dumas 
Social and Rehabilitation Service 
University of Wisconsin 
Madison, Wisconsin 53706 

Dr. Norman D. Ehresman, Director 
Center for Research in Vocational 
and Technical Education 
Box 8009, University Station 
University of North Dakota 
Grand Forks, North Dakota 58201 

Dr. Joseph R. Ellis 
College of Education 
Northern Illinois University 
De Kalb, Illinois 60115 

Mr. Walter J„ End 

Coordinator of Research and Evaluation 

Administration Annex S-502 

Sixth Street North 

West Palm Beach, Florida 33401 



o 
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Dr. Eldon E. Fahs 
Assistant to the President 
Manchester College 
North Manchester, Indiana 46962 

Dr. Christopher W. Flizak 
Department of Education 
State University of New York 
Fredonia, New York 14063 

Dr. Douglas E. Giles 
Supervising Principal 
Lemon Grove Public Schools 
Lemon Grove, California 92045 

Dr. Tim Gust, Director 
Research and Training Center 
in Vocational Rehabilitation 
University of Pittsburgh 
Pittsburgh, Pennsylvania 15213 

Dr. Jules Harcourt, Assistant Director 
Michigan-Ohio Regional Educational Lab. 
3750 Woodward Avenue 
Detroit, Michigan 48201 

Dr. Robert H. Hardy, Jr. 

Program Special i st 

Department of Program Development for 
Gifted Children 

Office of the Superintendent of 
Public Instruction (Illinois) 
Springfield, Illinois 26703 

Dr. Jack H. House 

Department of Educational Administration 
Ontario Institute for Studies 
In Education 
102 Bloor Street, W. 

Toronto 5 , Ontario 
CANADA 

Dr. M. Frances Kel 1 y 
Department of Higher Education 
State University of New York 
Buffalo, New York 14222 



Dr. Byron L. Kerns 
Dean of Students 
Mi 1 1 i k? n Universi ty 
Decatur, Illinois 62522 

Dr. Ralph E. Lundgren 
Director of Educational Research 
Office of the Superintendent 
of Public Instruction 
Springfield, Illinois 26703 

Dr. Raymond C. Man ion 
H. R r B. - Singer, Inc. 

P. 0. Box 60 

State College, Pennsylvania 16801 

Dr. Guy Marion 
2136 Chemin Ste-Foye 
Quebec 10, P. Q. 

CANADA 

Dr. Donald Melger 
Research and Development Center 
for Teacher Education 
University of Texas 
University Station - Drawer J 
Austin, Texas 78712 

Dr. Nick F. Muto 
Di rector. Title III 
1200 Westmoreland Avenue 
Syracuse, New York 13210 

Mr, Larry R. Nelson 
Director, Data Processing 
and Analysis 

Instructional Research Lab. 
University of Wisconsin 
Madison, Wisconsin 53703 

Dr. John R. Noak 
Assistant Director of Research 
Office of Public Instruction 
Springfield, Illinois 26703 
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f t 

Dr. 0. K. O'Fallon 
Director, Institute of Research 
and School Services 
College of Education 
Hoi ton Hal 1 

Kansas State University 
Manhattan, Kansas 6650*. 

Dr. Glenn E. Reel i ng 
Director of Institutional 
Research 

Jersey City State College 
Jersey City, New Jersey 07305 

Miss Barbara J. Rinker 
Director of Research 
l/D/E/A Office 

Marine Ci ty-Ward-Cott rel 1 High School 
Marine City, Michigan 48039 

Dr. Byron Keith Rose, Director 
Nortnern California PACE Center 
Chico State College 
Chico, California 95926 

Dr. J. Glen Scott 
Office of the Coordinator 
of Graduate Studies 
Ontario Institute for Education 
102 Bloor Street, W. 

Toronto 5, Ontario 
CANADA 

Dr. Norval C. Scott, Jr. 

Title III Center 
662 La Mesa Drive 
Salinas, California 93901 

Dr. Alan T. Seagren 
Coordinator of Laboratory 
Experi ences 

Mid-Continent Regional Educational 
Laboratory 

University of Nebraska 
Lincoln, Nebraska 68502 
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Dr. Richard L. Sheely 
Assistant Superintendent of 
Curriculum and Research 
23305 Chagrin Blvd. 

Cleveland, Ohio 44122 

Mr. Denis Tang’-nv, Executive Secretary 
Institute of Reheard. ' education 
Department of Education 
917 Mgr. Grandin 
Quebec 10, P. Q. 

CANADA 

Dr. Kenneth A. Tye 
Specialist in Program Planning 
Research Program l/D/E/A 
1100 Glendon Avenue - Suite 950 
Los Angeles, California 90024 

Dr. Ralph E. Walker, Assistant 
Superi ntendent 

Research and Information Service 
Unified School District #259 
428 South Broadway 
Wichita, Kansas 67202 

Dr. Eldon L. Wiley, Executive Director 
Southwestern Ohio Educational Research 
Counci 1 , Inc. 

1212 Oxford-State Road 
Middletown, Ohio 45024 

Dr. D. Parry Wi 1 son 
Director, Institutional Research 
Weber State College 
Ogden, Utah 84403 

Dr. Edgar N. Wright 
Director of Research 
Toronto Board of Education 
1 55 Col 1 ege St reet 
Toronto 28, Ontario 
CANADA 
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APPENDIX B 

AERA 1968 Research Training Presessions Program 

Presession Critique for Staff Members 
(For Directors, instructors and Assistants) 



Indicate your observation and judgement by checking each item 
in one column at the left. Items not applicable or not 
subject to your observation should be omitted. Be frank. 







1. 


Environment conditions 


3 


J 




a. 


Classroom spaces 


nr 






b. 


Work spaces 


1 


1 




c # 


Living quarters 




_3 




d. 


Teaching equipment, aids (chalk boards, public 
system, etc.) 


2 


1 




e. 


Resource material, library 


— — 


-i. JL 




f. 


Eat i ng faci 1 i t ie 5 






2. 


Part i ci pants 




h 




a. 


Appropriateness of academic backgrounds 




h 




b. 


Sufficiency of research experience 


~T 


1 




c. 


Willingness to work 


2 


2 




d. 


Intellectual curiosity 


3 


1 




e. 


Concern for applicability of techniques 


3 


1 




f. 


Aspi ration 




_3 




g- 


Immediate preparation for Presession 






3- 


Organi zat ion 


2 


_2 




a. 


Adequacy of notice to prospective applicants 


A 

Z 






b. 


Sufficiency of preplanning 


2 


2 




c. 


Smoothness of operation 


1 


3 




d. 


Adaptability to obstacles and feedback 




4 




e. 


Sensitivity to grievances 


JL 


Jl 




f. 


Adequacy of financial support 






4. 


Schedule 


1 


3 




a. 


Appropriateness of five days for the job 


2 


2 




b. 


Time spent efficiently 


2 


1 




c. 


Events sequenced appropriately 


1 


3 




d . 


Punctual i ty 


1 


3 




e. 


Balance between formal, informal affairs 


2 


2 




f. 


Quantity of discussions 


2 


2 




g- 


Quality of discussions 


3 


1 




h. 


Quality of formal presentations 


2 


2 




i , 


Unobtrusiveness of evaluation efforts 


— n_ — . 


3111 




j • 


Methods of evaluation 






5. 


Outcomes 


2 


_2 




a. 


Intended content was actually taught 


nr 






b. 


Increase in participant understanding 


2 


”2 




c. 


Improvement in attitude toward research 


3 


J 




d. 


Personal associations initiated 



o 
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6. In general was the Presession well organized? 

Yes - 4 



7. Were the facilities suitable for the activities which you had planned? 

If not, specify. 

Yes - 4 

8. Should Presessions be limited to the same hotel, or the same city, in 
which the annual meetings will be held? 

Yes - 2 
No - 2 

9. Were you to do the same assignment over, in what major ways, if any would 
you change your contribution? 

More discussion and application 
Develop more visuals 
Reorganize lesson material 

10. Do you wish that the Director had made firmer arrangements to assure 
participants and you of the staff opportunity to meet in pairs or small 
groups? 



No ” 3. They were quite adequate. 

11. Were the objectives you set for yourself during the Presession attained? 

Yes - 4 



12. Are you inclined to urge your colleagues to become staff members for such 
an institute or Presession? 

Yes - 4 



13. In what ways, if any, did you as a staff member benefit personally as a 
result of your participation in this Presession? 

Experience in organizing such a session - 2 responses 
Experience in presenting materials 
Planning and development of a simulation exercise 
Personal contacts 

Better understanding of participant^ problems and weaknesses 
of the Presession 



ANTHROPOLOGICAL FIELD METHODOLOGY IN THE STUDY OF EDUCATION 



Director 

Dr. Frank W. Lutz 
New York University 



STAFF 



Direc tor: 



Instructors : 



Assistants: 



Frank W. Lutz 
School of Education 
New York University 
4 Washington Place 
New York, New York 10003 

Howard S. Becker 
Northwestern University 
Department of Sociology 
Evanston, Illinois 

Louis M. Smith 
Washington University 
St. Louis, Missouri 

Laurence Iannaccone 
School of Education 
New York University 
4 Washington Place 
New York, New York 10003 

Earl Gaulke 
Graduate Assistant 
Washington University 
St. Louis, Missouri 

John Erratt 
56 Floral Terrace 
Tenafly, New Jersey 

William Hanigsbert 
2040 Touhy Avenue 
Chicago, Illinois 

Lawrence Kaplan 
65 Clover Lane 
Lido Beach, New York 
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PARTICIPANTS 



Doris Barclay 

2234 N. Buenos Aires Drive 
California State College at 
Los Angeles 
Covina, California 

B. Dean Bowles 
Department of Educational 
Adminis tration 
University of Wisconsin 
Madison, Wisconsin 

Richard Brandt 
203 Peabody Hall 
University of Virginia 
Charlottesville, Virginia 

Helen Burchell 
University of Pennsylvania 
3700 Walnut Street 
Graduate School of Education 
Philadelphia, Pennsylvania 

Emma M. Cappelluzzo 
School of Education 
University of Massachusetts 
Amherst, Massachusetts 

Courtney B. Cazden 
Harvard Graduate School of 
Education 

Longfellow Hall, Appian Way 
Cambridge, Massachusetts 

Robert L. Curran 
College of Education 
928 N.W. 36 Road 
University of Florida 
Gainesville, Florida 

Norman R. Dodl 

Department of Elementary Education 
College of Education 
University of Illinois 
Urbana, Illinois 

Pat Erickson 

431 Palmerston Boulevard 
Ontario Institute for Studies in 
Education 
Toronto 4, Ontario 



W. R. Fielder 
Claremont Graduate School 
Claremont, California 

F. Lincoln Grahlfs 

Sociological Resources for Secondary 
Schools 

503 First National Building 
Ann Arbor, Michigan 

Samuel Guskin 
Indiana University 
1725 East Hunter 
Bloomington, Indiana 

Frances F. Hanson 
Portland State College 
201 Old Main 
Portland, Oregon 

Robert W. Heller 

State University of New York at Buffalo 
4866 Kraus Road 
Clarence, New York 

Edward Hickcox 

Ontario Institute for Studies in Education 
102 Bloor Street West 
Toronto, Ontario 

Donald H. Kase 
North Bay Pace Center 
1005 Jefferson Street 
Napa, California 

Carl Lewis 

State University of New York at Fredonia 
Sociology Department 
Fredonia, New York 

William Lowe 
Cornell University 
Stone Hall 
Ithaca, New York 

Donald McCurdy 
University of Nebraska 
7901 E. Avon 
Lincoln, Nebraska 

Joseph McGivney 

Ohio State University 
9800 Kinnear Road 
Columbus, Ohio 
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Troy Me Ke Ivey 

State University of New York 
at Buffalo 
6805 Tuscany Lane 
East Ampherst, New York 

Mary Meehan 

Public Schools of Kansas City 

5304 Troost 

Kansas City, Missouri 



Jay Smink 

Department of Public Instruction 
Box 911 

Harrisburg, Pennsylvania 

LaVar Sorensen 
Salt Lake City Schools 
440 E. First South Street 
Sat Lake City, Utah 



Norma T. Mertz 
Eastern Michigan University 
507 Lafayette Towers East 
Detroit, Michigan 



William Streich 
Farmington Public Schools 
8 Mountain Road 
Farmington, Connecticut 



Vernon Moeller 

Department of Counselor Education 
College of Education 
Florida State University 
Tallahassee, Florida 

Norman Robinson 
Faculty of Education 
Simon Fraser University 
Burnaby 21, British Columbia 
Canada 

Mary Roose 

Iowa Psychologist for Washington 
County Board of Education 
Washington County Court House 
Washington, Iowa 

Monroe Rowland 
IDEA 

330 N. Main 
German town , Ohio 

Irvin Rub incam 
Bucknell University 
41 South Third Street 
Lewisburg, Pennsylvania 

Abilgail Sher 
Yeshiva University 
128 Fort Washington Avenue 
New York, New York 



Elwood Traylor 
Wichita State University 
2701 N. Pershing Court 
Wichita, Kansas 

Howard Van Sickle 
Mankato State College 
Mankato , Minne®"' ta 

H. A. Wallin 

University of British Columbia 

4620 W. 13 Avenue 

Vancouver 8, British Columbia 

Thurlow Wilson 
City University of New York 
535 E. 80 Street 
New York, New York 

Harry Wolcott 

Center for Advanced Study of Educationa 
Adminis tration 
University of Oregon 
Eugene, Oregon 

Frank Wood 

University of Minnesota 
103 Pattee Hall 
Minneapolis, Minnesota 



As indicated above the participants in this presession were quite a cross 
section. They ranged from those having long since completed their doctorates 
to those still working on it; from the west coast to the east coast and from 
Canada to Puerto Rico; from those having extensive experience in the methodology 
to those with none; from psychologists, anthropologists, ar 1 sociologists to 
curriculum specialists and educational administrators. Perhaps we were too prone 
to accept a cross section of those interested in studying education (too broad 
a category). We have suggested an alternative for future anthropological 
presessions . 



SCHEDULE 



The following schedule is the schedule actually followed in the presession. 
It will be noted that it differs from the tentative schedule first suggested. 
These modifications were made during staff meeting': held after the completion 
of the sessions. They were based upon the feedback of the participants and 
represented the consensus of the staff. The Director feels the staff was very 
helpful in this regard and the resulting schedule was an improvement over the 
original one. Although there was the feeling on the part of some participants 
that the changes resulted in a lack of organization, this was not the case. The 
result was a better presession. 



Actual Schedule 



Saturday, February 3 12:00 - 5:00 p.m. 

First Session (1:00-3:00 p.m.) 

Large Group A. An Overview of the Presession 

30 min. - Lutz 

B. Presentation of Work of Individuals 

1) Smith - 30 min. (Classrooms) 

2) Becker - 30 min. (Universities) 

3) Iannaccone - 30 min. (Schools) 

4) Lutz - 30 min. (Districts & 

School Boards) 

BREAK 

* Second Session (3:15-4:30 p.m.) 



Small Group 


A. 


Each of us met with the four 
discussion groups for fifteen 
minutes to discuss and inter- 
act with groups concerning first 
session presentations. 


Large Group 


B. 


View Film, "The Conference", take no 
notes; write up for next day 


Sunday, February 4 




9:00 - 5:00 p.m. 




Third Session (9:00-12:00 Noon) 




A. 


Discussions - What Happened in the 
Conference? (9:00-9:30) 


Large Group 


B. 


The Conference with notes or interview 
materials. (9:30-10:00) 




C. 


Data Collection-Analysis-Inference and 
Proof (10:00-11:00) - Becker 




D. 


Can Field Studies Test Hypotheses? 
What is the Meaning of Field Data? 



(11:00-12:00 Noon) - Smith 
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LUNCH 12:00 - 2:00 p.m. 



Fourth Session (2:00-5:00 p.m.) 



Large Group 


A. 


Verification (2:00-3:00) - Iannaccone 




B. 


Panel on above (3:00-3:30) Staff 


Small Group 


C. 


Use of scrambled cases (3:30-5:00) 


Monday , February 5 




9:00-5:00 p.m. 




Fifth Session (9:00-12:00 Noon) 


Large Group 


A. 


Discussions of scrambled cases 
(9:00-10:30)- Staff 


Small Group 


B. 


Staff's analysis of scrambled cases 
(10:30-12:00) - Smith 6c Iannaccone-A 6c ] 

Lutz 6c Becker - C 6c D 


T UNCH 12:00 - 2:00 p.m. 


Sixth Session (2:00-5:00 p.m.) 




A. 


Roles in Field Data Collection 
(2:00-3:30) - Lutz 


Small Group 


B. 


Small Group Di cuss ion (4:00-5:00) 


Tuesday, February 6 




9:00-5:00 p.m. 




Seventh Session (9:00-12:00 Noon) 


Large Group 


A. 


Tri systems model (9:00-11:00) - Lutz 


Small Group 


B. 


Small Group Discussion (11:00-12:00) 


LUNCH 12:00 - 2:00 p.m. 


Eighth Session (2:00-5:00 p.m.) 




A. 


Use of Interviews (2:00-5:00) - Becker 


Wednesday, February 7 




9:00-3:00 p.m. 




Ninth Session (9:00-12:30 p.m. 


Large Group 


A. 


Report Writing and the Morality Problem 
in Field Work (9:00-10:30) 


Small Group 


B. 


Informal Evaluation Discussions 
(10:30-12:00) - Staff 


Large Group 


C. 


Final Remarks (12:00-12:15) - Lutz and 
Staff 
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LUNCH 12:30 - 1:30 p.m. 

Tenth Session (1:45-3:00 p.m.) 
Discussion of kinds of research - Staff 

1. Historic and Demographic 

2. SES and social class 

3. Reputational 

4. Issue Analysis 



Original Tentative Schedule 

Saturday, February 3 12:00-5:00 p.m. 

First Session (1:00-3:00 p.m.) 

Large Group A. An Overview of the Presession - 30 min. 

Lutz 

B. Presentation of Work of Individuals 
Smith - 30 min. (Classrooms) 

Becker - 30 min. (Universities) 

Iannaccone - 30 min. (Schools) 

Lutz - 30 min. (Districts & School Boards) 




BREAK 



Second Session (3:15-4:30 p.m.) 



Small Group 



Large Group 



Sunday, February 4 



A. Each of us meet with the four discission 
groups for twenty- minute periods to 
discuss and interact with groups concerning 
first session presentations. 

B. View film, "The Conference", take no notesjJ 

write up for next day i 

9:00-5:00 p.m. 

Third Session (9:00-12:00 Noon) 



Large Group 



A. Data Collection - Analysis - Inference 

and Proof (9:00-10:00) - Becker 

B. Discussions - What Happened in the 
Conference? (10:00-10:30) 

C. Can Field Studies Test Hypotheses? 

What is the Meaning of Field Data? 
(10:30-11:30) - Smith 
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Small Group 



D. The conference with notes or interview 
materials (11:30-12:00) 

LUNCH 12:00 - 2:00 p.m. 



Large Group 



Individual Work 



Fourth Session (2:00-5:00 p.m.) 

A. Roles in Field Data Collection 
(2:00-3:00) - Lutz 

B. Panel on above (3:00-3:30) - Staff 

C. Use of scrambled cases (3:30-5:00) 



Monday. February 5 9:00-5:00 p.m. 

Fifth Session (9:00-12:00 Noon) 

Small Group A. Discussions of scrambled cases 

(9:00-10:00) - Staff 

B. Staff*s analysis of scrambled cases 
with overhead materials (10:30-12:00) 

LUNCH 12:00 - 2:00 p.m. 

Sixth Session (2:00-5:00 p.m.) 

Large Group A. Field Studies - Verification Studies 

Continuum (2:00-3:00) - Iannaccone 

Small Group B. Small Group Discussion (3:00-5:00) 

Each instructor will have an opportunity 
to pursue their large group presentations 
with each of four groups for 25 minutes 



Tuesday, February 6 9:00-5:00 p.m. 

Seventh Session (9:00-12:00 p.m.) 



Large Group 



A. Tri-systems model (9:00-11:00) - Lutz 

B. Reanalysis of the scrambled cases by 
Tri-system (11:00-12:00) - Iannaccone 



Large Group 



LUNCH 12:00 - 2:00 p.m. 

Eighth Session (2:00-5:00 p.m.) 

A. Other Data Collection Methods (2:00-5:00) 

1. Records -Historic -Demographic 

2. SES and Class Models 

3. Reputational 

4. Issue Analysis 

5. Field Experimentation 
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Wednesday, February 7 



9:00-3:00 p.m. 

Ninth Session (9:00-11:00 p.m.) 



Large Group 



Small Group 



Large Group 



A. Report writing and the morality 
problem in field work (9:00- 
11:00) - Panel 

LUNCH 11:00 - 12:00 NOON 

Tenth Session (12:00-3:00 p.m.) 

A. Evaluation (12:00-1:00) 

Form of Report Writing 

Theory and Models 
Data Collection 
Data Analysis 

Future Research and Verifi- 
cation Studies 

B. Informal Evaluation Discussion 
(1:00-2:00) - Staff 

C. Final Remarks (2:00-2:30) - Lutz 

and Staff 



MATERIALS DISTRIBUTED 



A. Twenty-two page bibliography of readings in descriptive research. This was 
very extensive and produced by Howard Becker. 

B. Notes of an interview done by H. Becker during his "teacher mobility study" 

(16 pages). This was used as a basis of discussion of the interview techni- j 

ques . 

C. A scrambled case of 10 pages developed by L. Iannaccone from his Whittman 
school work. It consisted of paragraphs describing individual teachers 
in a school. This was used as a "data source" allowing the participants 
to "build" the social structure of the school and discuss their analysis 
with the staff. 

D. The Tri-System Mode. A one-page graphic illustration of a model developed 
by F. Lutz from the work of Homans, Loomis, and General Systems, plus some 
of his own work. This served as a basis for a presentation on data collec- 
tion and data analysis and a group discussion of problems related to the 
topic . 

E. Tentative presession schedules and participant rosters were distributed. 

RESPONSES TO PARTICIPANT QUESTIONNAIRE 

Some of the questions of the questionnaire appeared to confuse the participants. 
For instance, "Was your room satisfactory" was answered by some using their s leepi ng 
quarters as a reference and others using their meeting room as a reference when answer-! 
ing the question. The Director encountered considerable difficulty in attempting to j 
tabulate the more or less free response answers to the questionnaire. A checked 
response would probably have been easier and more reliable to respond to. 
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Participant Evaluation 



Environment and Facilities 



1 a. 



b. 



2 a. 



b. 



To what extent did the relative availability or unavailability of 
books and journals interfere with or promote your attempts to 
master the content of this session? 

A Lot A Little None 

1 12 26 

To what extent did reproduced materials given to you by the staff 
improve matters? 

A Lot A little None 

12 20 3 

Did you feel that you lacked a "place to work", whether alone or in 
small groups? 

Yes No Indifferent 

7 34 1 



Was your room satisfactory? 

Yes No Indifferent 

31 6 1 



3 a. Which features of the meeting rooms were inadequate or not conducive 

to learning. 



Good 

4 



Bad Indifferent 

34 1 



Scheduling and Organization 

4 a. Was five days too long a period to leave your work at home for the 

purpose of attending this session? 

Too Long About Right Too Short 

20 22 

b. Was five days too short a period of time in which to learn much of 

the content of this session? 

Too Long About Right Too Short 

2 30 3 

5 a. Were you allowed enough time in which to pursue activities of your 

own choosing? 



Yes 

35 



No 

4 



79 



Would you have preferred not to meet in the evening after dinner? 

Did not meet in evening. The responses indicated it 
was the right thing to do. 

Would you have preferred more or fewer meetings per day than there 
actually were? Or was the number of meetings per day agreeable 
to you? 



Fewer or Shorter 
8 



More 

1 



OK 

30 



6 a. 



Were the individual lectures too long to sit and listen or take 
notes ? 



Yes 

10 



No 

20 



Occasionally 

11 



b. 



Were the lectures scheduled in an appropriate sequence? 



Yes 

30 



No 

7 



Not sure 
4 



7 a. 



Did you have sufficient opportunities to interact with other partici- 
pants ? 



Yes 

30 



No 

6 



8 a. 



Were the instructors too inaccessible or unapproachable so that you 
did not get the individual attention that you desired? 



Yes 

2 



No 

37 



b. 



Was it helpful to have graduate student assistants present' 



Yes 

16 



No 

20 



Didn 1 t know 
2 



9. 



Did the attempts to evaluate your progress and reactions during the 
session (and at this moment) interfere with your work here? 



Yes 

1 



No 

40 



10 . 



In general, was the presession well organized? 



Yes 

23 



No 

6 



Fairly Well 
8 



o 
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Content and Presentation 

11 a. Did the content of the lectures and readings presuppose far more 
previous training than you had? 

Yes No Didn't Know or OK 

1 39 

b. Should more or less training in these areas have been presupposed? 

More Less Didn't Know or OK 

10 6 12 

12. Was the content of the lectures and readings relevant to what you 

hoped to accomplish during the session? 

Yes No Sometimes or Usually 

23 3 7 

13 a. Were the lecturers stimulating and interesting? 

i 

All Some or Most Few or None 

18 20 2 

b. Were the lecturers competent to speak on the subject assigned 
them? 



All 

32 



Some or Most 

8 



Few or None 
1 



Were the lecturers well prepared? 



All 

29 



Some or Most 
9 



Few or None 
2 



14. 



Were you disappointed in any way with the group of participants? 



Yes 

11 



No 

27 



Didn't Know 
1 



15. 



16. 



If you had it to do over again, would you apply for this presession 
which you have just completed? 



Yes 

11 



No 

8 



If a presession such as this is held again would you recommend to 
others that they attend? 



Yes 

31 



No 

8 



Didn't Know 
1 



o 



17 , 



18 . 



Do you anticipate maintaining some sort of contact with at least 
one of the presession staff? 



Yes 

27 



No 

12 



Didn ' t Know 
2 



Do you feel that AERA is making an important contribution to 
education by sponsoring presessions such as this one? 



Yes 

37 



No 

1 



19 



20 , 



Do you feel that anything has happened during these five days to 
make it more likely that you will leave your present position of 
employment ? 



Yes 

2 



No 

32 



Is it likely that you will collaborate in research with someone 
else attending this presession (other than those you already 
were likely to collaborate with)? 



Yes 

10 



No 

24 



Perhaps 

3 



21 



Do you think that the staff should feel that i has accomplished 
its objectives during this five-day presession? 



Yes 

28 



No 

4 



Partially 

5 



PARTICIPANTS COMMENTS 



In addition to the questionnaires supplied by AERA, we asked the partici- 
pants to write a page indicating their feelings about this presession. The 
following paragraphs are taken from these comments. Generally the participants 
appeared satisfied with the presession. Most provided not only commendatory 
comments but also helpful suggestions. Some were openly critical of certain 
aspects or individuals. 

For some the presession provided experiences which will directly influence 
their research. Comments such as, "...terribly helpful at this point in my own 
work", -- "I have been exposed to an area that I previously ignored", -- "I 
picked up many ideas for immediate and practical use in my own work", --"...may 
lead to some changes in research I am doing", "...this session will be useful for 
my research" --- indicated that the presession will influence research in education. 
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Other comments suggested possible improvements for future presession in 
anthroplogical methods. "The presession should assume a higher level of general 
intellegience" and "it may have been better to separate into two groups" express 
the correct notion that there were great differences in the amount of previous 
experience in the use of the methodology evidenced in the group. Perhaps a "team 
teaching" organization, using only a few sessions with the total group and two 
programs (one advanced and one beginning) based on the previous experience of 
the participant was needed. The suggestion made several times "data collection 
in the field could have proven valuable" indicates a need on the part of the less 
experienced participant. Comments like. "some material too elementary" and "more 
penetration should have occurred" indicates a more experienced need for participants. 
Other comments like "interesting and informative session, but it did not serve my 
needs" and "some good and some bad", indicated the differing needs and experience 
of the participants. 

Some comments were sharply critical of indidivual staff members. Each staff, 
member was named as not being effective but each was also specifically named as 
being "the best in the session". Again the differences of the participants is 
pointed out. That the staff, through its diversity, was able to reach different 
participants is held to be an asset and was expressed by some participants — 

"I felt that the staff was remarkably open and frand" — "very competent staff" 

-- "well-balanced staff in terms of previous work and style of working with the 
group". 

Finally most participants felt the presession had been a good one. The 
following quotes express such a feeling. Regarding che program --"I can't 
conceive of a more appropriate sequence" -- "highly stimulating" -- "very 
helpful session" — "certainly worth attending" -- "well organized - the 
concepts discussed were of value" -- "great idea - especially as a counter 
to the general mood of AERA" — "informative, interesting, helpful" -- "a 
very worthwhile five days". As to personal benefit, participants indicated, 

"the experience was rewarding and stimulating" -- "on the whole just what I 
wanted" — "I received precisely what I came for". The above three paragraphs 
represent short excerpts from reports by participants. No two comments came 
from the same participant nor did most participants only praise or criticize. 

Most criticisms were very constructive. On the whole, these paragraphs repre- 
sent the feelings of those participants (37 of 48) who wrote a comment page. 

These will prove very valuable in improving future presessions on anthroplogical 
methods should AERA approve such. 



RESPONSES TO STAffr QUESTIONNAIRE 



Staff Questionnaire* 



Comm 



Sat 



Unsat 



1. Environmental conditions 

a. Classroom spaces 

b. Work spaces 

c. Living quarters 

d. Teaching eqiupment, aids 

e. Resource material, library 

f. Eating facilities 



4 



3 

7 

8 

4 
6 
8 



5 

1 



2 



*There were four instructors and four graduate! assistants responding, thus a 
possibility of eight responses. . 
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Comm 


Sat 


Unsat 


Participants 








a. 


Appropriateness of academic background 




8 




b. 


Sufficiency of research experience 




8 




c. 


Willingness to work 




8 




d. 


Intellectual curiosity 


2 


6 




e. 


Concern for applicability of techniques 


2 


6 




f. 


Aspiration 




8 




g- 


Immediate preparation for presession 




6 


2 


. i 

Organization 








a. 


Adequacy of notice to prospective 










applicants 




8 




b. 


Sufficiency of preplanning 




6 


2 


c. 


Smoothness of operation 




7 


1 


d. 


Adaptability to obstacles and feedback 


8 






e. 


Sensitivity to grievances 


8 






f . 


Adequacy of financial support 


6 


2 




Schedule 








a. 


Appropriateness of five days for the job 




7 


1 


b. 


Time spent efficiently 




7 


1 


c. 


Events sequenced appropriately 




8 




d. 


Punctuality 




8 




e. 


Balance between formal, informal affairs 




8 




f. 


Quantity of discussions 




8 




g- 


Quality of discussions 


1 


7 




h. 


Quality of formal presentations 


2 


6 




i. 


Unobtrusiveness of evaluation efforts 


5 


3 




j- 


Methods of evaluation 


1 


7 




Outcomes 








a. 


Intended content was actually taught 


5 


3 




b. 


Increase in participant understanding 


6 


2 




c. 


Improvement in attitude toward research 


1 


7 




d. 


Personal associations initiated 


4 


4 





In general was the presession well organized? 

Generally "yes" — but ability to get staff together before the 
presession would help a lot. 

Were the facilities suitable for the activities which you had planned? 
If not, specify. 

Generally "yes" — the hotel did what it could, but small 
group flexability was a hinderance to operation. 

Should presessions be limited to the same hotel, or the same city, in 
which the Annual Meetings will be held? 

We had hoped to hold ours in the convention hotel for the 
convenience of the participants only to find a different 
presession hotel from regular session hotel. Why? 
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9. Were you to do the same assignment over, in what major ways, if 
any, would you change your contribution? 

(See recommendations) 

10. Do you wish that the director made firmer arrangements to assure 
participants and you of the staff opportunity to meet in pairs or 
small groups? 

Some "Yes" Some "No" 

11. Were the objectives you set for yourself during the presession attained? 

All "Yes" 

12. Are you inclined to urge your colleagues to become staff members for such 
an institute or presession? 

All "Yes” 

13. In what ways, if any, did you as a staff member benefit personally as a 
result of your participation in this presession? 

All felt they benefited from the experience of interacting 
both with the participants and with other staff members. 



EVALUATION 

It was decided by the staff that mastery tests were not desirable in such 
a presession and we indicated in our proposal that none would be given. We 
followed this notion. Participants were given the opportunity to evaluate 
themselves and discuss their work in the following ways: 

1. Field observation and note taking 

2. Data analysis from field data 

3. Analysis of interview data 

4. Participants questionnaire 

5. Informal, small group evaluation of the presession 

From the above and the comments of the staff, the following suggestions are 
derived for future presessions in the anthropological method in education^ 

1. ill the staff and many of the participants recommend that AERA 
continue to sponsor presessions in anthropological methods. 

2. It has been suggested that Division D (Methodology) take some interest 
in the anthropological or descriptive methodology of educational research. 
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3. Any future attempt at such a presession should divide the total 
group into experience levels and research interests (i.e., administra- 
tion, classroom, educational politics). They should then utilize staff 
election and team teaching methods in order to meet the varying needs 
of the participants. 

a) Field experience should be part of the beginners session 

b) Extensive share research should be part of the experienced 
session 

c) Both sessions should receive a "package" of before-session 
"homework" 

4. A "happy hour" session should precede the actual presession. 

5. Coffee breaks should be a planned part of the sessions (perhaps at 
a fee of $5 or $6 could be charged to cover cost of #4 and 0 = 5 ) . 

6. An individual whose formal course work trained him in anthropology 
should be included. (The four staff members had_ used the anthropological 
method extensively to study education. But rone were trained as anthro- 
pologists.) 
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PRESESSION IV 



NONPARAMETRIC METHODS AND RELATED POST HOC PROCEDURES 



Director 

Dr. Leonard A. Marascuilo 
University of California at Berkeley 

. o 
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PRESESSION OBJECTIVES 



The objectives of the Presession on Nonparametric Methods were stated 
in simple operational terms. They were designed to satisfy some of the 
needs of behavioral scientists whose duties included research in an 
educational setting and not in the development of new statistical procedures 
and methods. Since more and more educators are becoming involved in research 
and since the number of nonparametric methods available constantly increases, 
it becomes essential that active educators and behavioral researchers be 
informed and kept up to date on the advances made in this important area of 
research methodology. Consequently, this Presession was planned for researchers 
in education who had minimal backgrounds in statistics and was executed with 
their assumed needs in mind. 

A printed statement of the primary goals and objectives of the Presession 
was made available to potential participants in the announcement "Call for 
Applications to the AERA 1968 Research Training Presessions" which appeared in 
the Educational Researcher . The specific announcement read as follows: 

"The course content will be: 

"Content and Objectives: The course content will be restricted to 

nonparametric techniques extensively used in education and the behavioral 
sciences. The first part of the course will treat nonparametric tests designed 
for qualitative variables, such as the Sign Test, the Median Test, and the 
Irwin- 1* isher Exact Test. The second part of the course will emphasize non- 
parametric tests based on ranking procedures: the one sample Wilcoxon Matched- 

Pairs Test, the Wilcoxon Two Sample Test, the Krusal-Wallis extension of the 
Wilcoxon Test, the Friedman Test for Repeated Measures, and the corresponding 
post hoc procedures, as well as Spearman's Ranked Correlation Coefficient and 
Kendall's Coefficient of Concordance. The primary objective will be to assist 
the participants in applying the techniques of nonparametric statistics to the 
solution of problems of hypothesis testing and estimation commonly occurring 
in educational research. 

"Anticipated Audience: This session will be open to persons whose 

responsibilities include the design of educational research studies and the analy 
of research data. The course is primarily intended for educational researchers 
whose commitment is to substantive areas other than that of statistics." 

Since the primary objective of the Presession was to assist the participants 
in applying the techniques of nonparametric statistics to the solution of 
problems of hypothesis testing and estimation commonly occurring in educational 
research, it was hoped that at the end of the presession the participants would 
be able to: 

a. Explain to other researchers and educators the rationale behind the 
nonparametric tests commonly employed in behavioral research. 

b. Select the "best" nonparametric test for a specific situation or 




I 
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experimental design in both laboratory and school settings. 

c. Perform nonparametric tests on data collected in educational research 
studies of observational or experimental nature. 

d. Conduct appropriate post hoc comparisons on hypotheses rejected 
by nonparametric procedures. 

e. Apply and extend the nonparametric techniques learned to specific 
problems in educational research for which textbook examples are not 
available . 

f. Read with understanding the current literature on nonparametric 
methods appropriate for research in education. 

g. Read with understanding current education research studies and reports 
that use nonparametric statistics. 

h. Direct other researchers in the use of nonparametric methods. 

STAFF 



The Director and his staff are as follows: 
Director: 

Dr. Leonard A. Marascuilo 
Department of Education 
University of California at Berkeley 
Berkeley, California 

Instructional Staff: 

Dr. Ma rye lien Me Sweeney 
School of Education 
Michigan State University 
East Lansing, Michigan 

Dr. Douglas Penfield 
Department of Education 
University of California at Berkeley 
Berkeley, California 

Assistants : 

Mr. Michael Subkovialc 
School of Education 

State University of New York at Buffalo 
Buffalo, New York 

Mr. Larry Leslie 
Department of Education 
University of California at Berkeley 
Berkeley, California 



0 
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The Director of the Presession selected his staff members from associates 
whose proficiencies and competencies in statistics and administration were well 
known from previous work and study experience. 

Both instructors were, at one time, graduate students in the Statistics 
and Measurement Program under the guidance of the Director. Furthermore, 
both instructors had taken theoretical courses in Nonparametric Statistics 
in the Department of Statistics at the University of California at Berkeley. 

As a result, they were quite knowledgeable in the exciting and major advances 
in nonparametric methods that have taken place in that active department. 

Many of the new techniques which they had learned were presented for the first 
time to behavioral researchers at this Presession. In addition, both instructors, 
while graduate students, had considerable teaching experience in the statistics 
courses required of all Ed.D and Ph.D. candidates in the Department of Education 
at the University of California at Berkeley. Student reports on the quality of 
teaching performed by both instructors has always been of the highest kind. 

Thus, both staff members were selected because they were available, they were 
known to be excellent teachers, and they possessed an excellent grasp and under- 
standing of the materials to be taught at the Presession. 

The assistants were also selected on the basis of known competence. One 
of the assistants, who also conducted some of the Presession teaching, served 
as the Director’s Teaching Assistant for a graduate course in Nonparametric 
Methods taught during the summer of 1567 at the State University of New York 
at Buffalo. His performance as a Teaching Assistant was excellent. The remaining 
assistant was selected because of his proven abilities as an administrator and 
coordinator of educational projects. He was known to the Director through a 
course that he took in Nonparametric Statistics taught by the Director at the 
University of California at Berkeley. 

Without doubt, the Director’s expectations concerning the performance of 
his staff during the Presession were well met. Each member of the staff performed 
at a level commensurate with their known proficiencies and competencies. Their 
skills were effectively used. 

SELECTION OF THE PARTICIPANTS 



Applications were received through the mail from Dr. Gene V. Glass of 
the University of Colorado. They were screened and evaluated by the Presession 
Director and Dr. Douglas Penfield of the Instructional Staff. Fifty-nine 
applicants were admitted to the Presession. Forty-nine actually attended. 
Criteria used for selecting the participants were as follows: 

I 

1. Doctorate . Following the advice of Dr. Gene Glass, it was decided 
that applicants not holding a doctorate would no be admitted since, in most 
cases, they would have had the opportunity to develop skills in nonparametric 
methods through regular doctoral programs at their local institutions. However, 
it was also recognized that many small schools and research agencies might not 
offer courses in nonparametric statistics and so consideration was also made 
for some students who did not hold a doctorate. 
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2. formal training in statistics . Because of the level of instruction 
contemplated for the Presession on Nonparametric Methods, it was necessary 

to assume that every participant had training in statistics beyond a first 
course. Therefore, students who had had only one course were, generally, 
not admitted. In the announcements, never made available to potential 
participants, it was suggested that applicants should have familiarization 
with the binomial and hypergeometric distributions along with knowledge of the 
one-way analysis of variance and the related post hoc comparison procedures of 
Scheffe'’. Most of the participants who did appear for the Presession did not 
have this minimal training. 

3 . Estimate of potential to contribute to education research activities . 
While this was to be given considerable weight, it was not necessary, since 

it appeared that Criterion Two eliminated, on a self-selection basis, many 
researchers who would have applied but who would not have been accepted. 

NOTIFICATION TO APPLICANTS OF THEIR ADMISSION TO THE PRESESSION 



Applicants accepted to participate in the program were informed of 
their acceptance by means of the following letter sent them at the end of the 
Fall Quarter just prior to the Christmas vacation period: 

It is a pleasure to inform you that your application has 
been accepted to the AERA Presession on Nonparametric Methods. 

The presession is being held at the Chicago Sheraton between 
8:30 a.m. on Saturday, February 3 to 5:00 p.m. on Wednesday, 

February 7. For this program the Sheraton has offered special 
rates for presession participants. In order to take advantage 
of these presession rates, it is advisable that you make hotel 
reservations as soon as possible. Since instruction will begin 
promptly at 8:30 a.m., it is suggested that you arrive at the 
Sheraton on February 2. 

The course will use as a text Distribution Free Statistical Test 
by James Bradly, which will be available for purchase at the 
first presession meeting. This test is a U.S. Department of 
Commerce publication and retails for approximately $3. In 
addition to the text, prepared handouts will be distributed to 
the participants. 

My staff and I will do all we can to make this presession match 
your expectations. We are looking forward to meeting you and of 
being given the opportunity to discuss mutual problems and research 
in education. If you have any questions regarding the presession, 
please write to me at Room 4511, Tolman Hall, University of 
California, Berkeley 94720. 

DESCRIPTION OF THE PARTICIPANTS 

Of the 59 participants admitted to the Presession, 49 appeared on 
the opening date. All 49 remained with the Presession throughout its 
duration. On the first day, following the first lecture, two participants 
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felt that they were unprepared and asked to be shifted to another presession 
being held in the hotel at the same time. An unsuccessful attempt was made 
to transfer them. As a result, they remained with the group. In the end, 
they reported that they were pleased that they had not changed to another 
presession since they had gained new insights in statistics and nonparametric 
methodology which could be used in their own research and in the evaluation 
of the research of others. 



The 49 participants who took part in the Presession are described in 
a summary of the biographical information collected on the application forms. 



1. Sex of the Participants. 
Sex 



Frequency 



Percent 



Male 


35 


71 


Female 


14 


29 




49 


100 


might have been expected. 


the bulk of the 


participants were 


Age of the Participants. 


Age 


Frequency 


Percent 


20-29 


6 


12 


30-39 


23 


47 


40-49 


17 


35 


50 and over 


3 


6 




49 


100 



Most of the participants were under forty, which suggests that the 
potential for future research in the entire group is relatively high. It is 
also believed, though it is only a subjective feeling, that the younger members 
gained the most from the Presession. They seemed to be the most enthusiastic, 
the most questioning, and the most eager to learn and increase their knowledge 
of nonparametric methods. 

3. Institutional Affiliation. 



Type of Institution 


Frequency 


Percent 


College or University 


33 


67 


Research Center 


10 


20 


Other 


6 


13 




49 


100 
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Most of the participants held professorial positions at their respective 
teaching institutions. Some of them were responsible for the teaching of 
statistics and research design and planned to use what they learned in their 
teaching. A few of the participants were concerned with reading important 
research in their own areas of inquiry and felt that a knowledge of nonparametric 
statistics would help them in understanding and improving their own evaluations 
of what others have done and written. 



4. Attendance at previous AERA Presession. 



Previous Attendance 



Frequency 



Yes 

No 



14 

35 

49 



Percent 

29 

71 

100 



For most of the participants, this was a first exposure to AERA 
presessions. From their general comments, they seemed well satisfied and 
on the basis of this first experience were quite anxious to attend another 
presession on topics of personal interest, either next year or in the near 
future . 

5. Years since Doctorate (for those who had Doctorate.) 



Years 



Frequency 



Percent 



1 5 

2 3 

3 9 

4 1 

5 6 

6 to 9 8 

10 or more 5 



37 



14 

8 

24 

3 

16 

22 

13 

100 



Nearly 50% of the participants had obtained their doctorate in the past 
five years. This characteristic correlated with their relatively young ages. 
Since many of the participants reported that basic courses in statistical 
inference were poorly taught or not taught at all at the institutions where 
they earned their doctorate, it follows that Schools of Education are not 
adequately preparing their candidates for research. As such, it must follow 
that the presession program of AERA is of immense value and should definitely 
be continued and expanded. 



6. Courses in Statistics. 



Number 



1 

2 

3 

4 

5 

6 



requency 


Percent 


5 


10 


11 


22 


14 


29 


8 


16 


2 


4 


9 


19 


vr 
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Since more than a beginning course in statistics was a requirement for 
admission, one might think that most participants would have been prepared 
or qualified to take this course. However, as was soon learned, number of 
courses is not a very satisfactory criterion variable for measuring under- 
standing and sophistication. Unexpectedly, the class was quite heterogeneous 
and less homogeneous than planned for. Even though 68 % of the participants 
had three or more courses in statistics, less than one-fourth knew of Scheffe's 
Theorem or Tukey’ s post hoc multiple confidence interval procedures, truly two 
of the most important procedures for behavioral research. 

7. Allocation of duties between teaching and research (for those who responded 



to this item) . 
Amount of time 


Teaching 

Frequency 


Percent 


Research 

Frequency 


Percent 


0% to 24% 


5 


16 


6 


13 


25% to 49% 


7 


22 


11 


24 


50% to 74% 


9 


28 


9 


20 


75% to 100% 


11 


34 


19 


43 




32 


100 


45 


100 



It is quite interesting to note that 39 of the 49 participants spend more 
than 25 % of their time in research, and yet their statistical training is clearly 
not sufficient for the research they are doing. Without doubt, this course 
was necessary for them, and perhaps even more important, it served to teach 
them more basic concepts in statistics than were expected either by the 
participants or by the instructional staff. It thus seems that more incidental 
learning occurred than was anticipated. 

8. Articles Accepted in Refereed Journals. 



Number 



Frequency 



Percent 



0 21 

1 6 

2 9 

3 3 

4 1 

5 1 

6 or more 8 



49 



43 

12 

18 

6 

9 

4mm 

2 

17 

100 



More than 50% of the participants had at least one article accepted for 
publication in a refereed journal. Considering the number of years since 
the doctorate, this suggests that the group did have some commitment to 
research and publication and because of their voluntary attendance at this 
Presession, the commitment must be quite high. 



9. 



Number of articles and reports written. 
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Number 

0 to 9 
10 to 19 



Frequency 

38 

11 

49 



Percent 

78 

22 

100 



Considering the number of years since the doctorate, this represents 
a fairly large number of reports and articles. 

10. The number of research projects funded by USOE, NIMH, or other 
granting agencies. 



Number 

0 

1 

2 

3 

4 

5 

6 



Frequency 

29 

10 

4 

3 

1 

1 

1 

49 



Percent 

59 

20 

8 

6 

2 

2 

2 



100 



As these statistics suggest, the participants are quite involved with 
research. 

The participants came from 23 states, Ontario, Canada, and Puerto Rico. 
The state? and nLber of participants each contributed are as follows: 



State 


Frequency 


Area 


Alabama 


1 


South 


Arkansas 


1 


South 


California 


2 


West 


Colorado 


1 


West 


Connecticut 


1 


East 


Illinois 


2 


Middle West 


Indiana 


2 


Middle West 


Kansas 


1 


Middle West 


Kentucky 


1 


South 


Maryland 


1 


South 


Massachusetts 


2 


East 


Michigan 


2 


Middle West 


Minnesota 


1 


Middle West 


Mississippi 


1 


South 


New York 


T 


East 


Ohio 


7 


Middle West 


Oklahoma 


1 


Middle West 


Oregon 


1 


West 


Pennsylvania 


1 


East 


Texas 


1 


South 


Virginia 


1 


South 


Washington 


2 


West 


Wisconsin 


b 


Middle West 
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Twenty of the 49 participants were from the Middle West. Since most 
of them were paying their own expenses, this is understandable. This also 
suggests that perhaps AERA should attempt to locate and obtain outside funds 
which would enable more people to attend future presessions. Certainly 
proximity to the presession site is an important factor in deciding 
whether or not an individual is going to apply for attendance to courses 
of this nature It is worth noting that the Director is conducting a two-day 

Eduction V” Non P^ ametrlc Methods in March of this year for the California 
Education Research Association. At the time of this writing, 25 educators 
have applied. All are from California. 



The nam-s and addresses of the participants are as follows: 



Auria, Carl 



Clark, Hi chard 



bureau of Educational Research Department of Ed. 
Ken rotate University State Univ. of N. 

ent * 0hl ° Albany, New York 



Psych. 

Y • at Albany 



Avital, Jhmvel M. 

Ontario Institute for 
Studies in Education 
102 Bloor West 
Toronto, Ontario, Canada 

baich, Henry 
School of Education 
University of Portland 
Portland, Oregon 

Capo de Rivero . Lillian 
Division of Evaluation 
Department of Education 
University of Puerto Rico 
Rio Piedras, Puerro Rico 



Clegg, Ambrose, Ai, Jr. 

4021, 97th Avenue, S. E. 
Ilercer Island, Washington 

Damico, Sandra 3. 

Ohio ^ 3du c at i. on A s so c i. at ion 
225 East* Broad Street 
Columbus, Ohio 

D r a rn ri gh t , Ru 3 s o 1 0 . 

Department of Education 
Bowling Green 'State University 
Bowling Green, Ohio 

Ferris, Panford J. 

39764 Cos t a Way 
Fremont , Califo rnia 



Fink, Ruth W. 

67 IO Gatos Mills Blvd. 

Gates Mills, Ohio 

Gadzolla, Bernadette M. 

312 Algoma Blvd., Apt. 301 
Oshkosh, Wisconsin 

Cill, Newell T* 

College of Education 
Eastern Kentucky University 
Richmond , Kentucky 

Gladden, Richard K. 

545 Dartmouth Drive 
King of Prussia, Pa. 



Johnson, Cecil L. 

#35 W c st T re n t A ve n ue 
Spokane , Wash ington 

Keepes, Bruce D. 

Palo Alto Unified School Dist * 
25 Churchill Avenue 
Palo Alto, California 

Keith, It. Virginia 
Faculty of Education 
University of Ottawa 
Ottawa, Ontario, Canada 

(Kliebhan) Sister M. Camille 
6 #01 North Yates Road 
Milwaukee, Wisconsin 
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Gna/ ;ey W.i 1 li am J . 

Illinois State University 
Psychology Department 
N o rn 1 a .1 , 1 1 1 i n o i s 

Gonriella, Carmella 
Boston University, Sargent 
College 

University Road 
Boston, Mass. 

Gozali, Joav 
Yeshiva University 
55 Fifth Avenue, Hm* 1121 
Now York, Mew York 

Graf, Holland W. 

3520, 30th Avenue 
Kenosha, Wisconsin 

Gulo, E. Vaughn 
N o r t h e as t e r n IJ ) i i v e r s i ty 
Boston, Mass. 

Hick, Thomas L. 

C h i 1 d 3 1 u dy C e n t e r 
3 1 at o U n i vers i ty 
Hew Halts, Hew York 

111 II, Russell A. 

Address Unknown 

Irwin, Claire G. 

369 Education Building 
College of Education 
Wayne State University 
Detroit, Michigan 

hiatus, Walter S. 

J« M. Patterson Building 
University of i Maryland 
C o 1 1 c g e , Pa rk , ? 1 a ry 1 an d 

Ku r r -a y , F ran k T> . 

Go pa r »..mo ] 'i t o f Ed • Psych. 

U n i v <: > r s i t y op 1 1 .1 n n o so t a 
iii iinoapoiis, r'linnosota 

Nelson , Randolph ,J. 
Department oB Ed. Psych. 
University of Connecticut 
Storrs, Connecticut 



Kohler, Emmett T. 

Box 911 

State College, Mississippi 

Krogstad, Roland J. 

2?1$ Regent Street 
Madison, Wisconsin 

Lewis , Leslie 

Southeastern State College 
Durant, Oklahoma 

Liechti, Carroll D. 

Research M Information 5er« Div. 
Unified’ School Dist. 259 
W i c h i t a , K an s a s 

Lopez-Machado , Gregorio 
r r i R- 15, V i 3 1 a F o n t an a 
Carolina, Puerto Pico 

Lu n d s t eon , S a r a \ ' . 

University of Texas 
College ol Education 
Austin, Texas 

Marlowe, Byron U. 

225 E as t • B ro a d S t r c c t 
Columbus, Ohio 

M c C o rm i c k , Flor e n c e R . 

South Central Legion Educational 
Laboratory 

Little Rock, Arkansas 



Rusch, Reuben R. 

State Univ. of ?!. Y. at Albany 
22$ Education Building 
Albany, Mew York 

Silverman, Toby R. 

Lexington School for the Deal 
New York, New York 

Slaiche r t , w i 1 1 i am 2 1 . 

School oB Education 
University of Denver . 

Denver, Colorado 
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Null, Eldon J. 

Department oT Education 
Pu rd ug U reivers! by 
Lai' ay ( 3 1 \j e , J n di an a 

2iggo, Free 1 L. 

Doijartinont of Education 
bowling Green- State Univ. 
Bov/ling Green, Ohio 

Itajpal, Puran Lai 

Si rei j. wo o < S Drive 
Fredonia, Wow York 

I t.o bo It , Mary F, 

303 _ North Hamilton Street 
Ypsilanti , Michigan 

Roll re r, Janet i i. 
ivonr, State University 
Dept, oi ORfice Adminis* 
Kent, Ohio 

Rosemier, Robert A, 

College oP Education 
Northern 111:1 no is University 
DeKalb, Illinois 



Smith, Arthur E. ' Mi' ' 

S t , Ma ry Mg- Co 1 1 e go 
Notre Dame, Indiana 

Thomas, Margaret J. 
Presbyterian Board of Chris- 
tian Education 
Box 1176 - 

Richmond, Vi rgi nia 

Watkins, Jan.es Poster 
P . 0 • Box 8 69 

First National Bank Building 
Auburn, Alabama 

Wilson, Ellen. 

Clarke Institute op Psychiatry 
250 College Street 
Toronto, Ontario, Canada 



PROCEEDINGS 



A. Schedule of Events. 



Participants arrived at the Sheraton Chicago Hotel during the dav 
proceeding the commencement of the Prosession. Arrangements for travel and 
i unding of thair stay were left ter each participant. 

staff oWbJBYr Schedule w f If dra "“ U P fa y Director and his Instructional 
.caff pnoi to the Presession. The schedule was closely followed except that 

the discussion sections were abandoned and replaced with lectures. It was 
thriectuLs^proc^ded!'" 68 “ " ^ qU6Sti ° nS * the P-^icipants as 

B • Presession Structure . 

, Instruction was conducted by means of four ninety-minute lectures 
per day. . Prepared handouts accompanied each lecture. Related readings 
were assigned in the text, Distribution- Free Statistical Tests by James 
V. Bradley, WADD Technical Report, 60-661 (Office of Technical Services 
U.S. Department of Commerce, Washington, D.C., $3.00) which participants 
pure ased at the first lecture of the Presession. Reprints of relevant 
journal articles were made available to the participants. Originally, partici- 
pants were expected to write short group reports on exercises solved durin- 
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planned informal discussion periods. These were to have been used for 

participant evaluation. This goal was dismissed when it was learned that 

not all participants were sufficiently prepared for the material to be 
presented. 



THE COURSE OUTLINE 

The course outline was built around clusters of nonparametric tests 
and related experimental designs. Consideration was also given to a 
structure whereby new material was built on procedures described earlier. 
The course content was as follows: 

I. Tests of Hypotheses Involving Proportions: 

, ^ * TIie course began with the Fisher Exact test for the equality 

o± tne parameters of independent binomial ly distributed random variables. 

2. As a special case of the Fisher Exact test, the Median test 
was presented and illustrated. 



3 \ The large sam P le extension of the Fisher Exact test to the 
amiliar Chi-square test was made and examples were used to illustrate 
the procedures . 



4. The Median test was extended to more than two samples and 
examples ftom educational research were considered. 

5. Post hoc procedures for the. Chi-square test and the large sampl 
Median test were derived and illustrated. 

II. Tests of Independence for Contingency Tables: 

1. This presentation began with tests for statistical independence 
for two dichotomous variables. These were derived, the phi coefficient was 
introduced, and examples were presented. 

2. Tests for interaction across contingency tables and associated 
post hoc procedures were illustrated and discussed. 

III. Tests on Correlated Observations (Repeated Measures Designs): 

1. The Sign test was introduced and illustrated. 

, -k , 2 ’ ® P ! C f al cases of the Si S n test > Stuart's tests for trend, were 
described and their use was demonstrated. 

3. Spearman's rho and the Wilcoxon Matched-Pairs Signed Ranks test 
were treated in one lecture with respect both to theory and real data. 

4. Kendall's coefficient of concordance and the Friedman test 
were illustrated and derived. 



5. Post hoc comparisons for the Friedman test were discussed and 
applied to an experimental study. 
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6. Cochran's Q tost and associated post hoc procedures were obtained 
and the McNemar test as a special case of the Cochran Q test was illustrated. 

2 

7. Rank analog to Hotelling's T and the corresponding post hoc pro- 
cedures were introduced and an example of their use was presented. 

IV. Tests of Independent Observations from Two or More Samples: 

1. The Wilcoxon test (Mann- Whitney) was introduced and an application 
of the Wilcoxon test to block designs was presented. 

2. Three forms of the Normal Scores test were described and illustrated. 

3. The Kruskal-Wallis test as an extension of the Wilcoxon test was 
made and post hoc procedures associated with the Kruskal-Wallis test were 
presented and illustrated. 



r 



TIME SCHEDULE 



adopted : 



Day 



The planned schedule was ignored and the following schedule was 



1 



Tine 



8:30 - 10:00 

10:30 - 12:00 
Is 30 - 3; 00 



3:30 

8:30 

10:30 



5:00 

10:00 

12:00 



1:30 - 3:00 
3:30 - 
8:30 - 10:00 



10:30 - 12:00 



Topic 



o b i o ctivos and r- 



Discussion of clasi 
view ol classical statistical methods. 

Continuation of review. 

The Binomial and Hypergeometric distri- 
bution. ' ; 

Fisher exact test. 

Median test and Chi-square test. 

Extension of the Median test to K 
samples. 

Post hoc procedures for the Chi-square 
and Median tests. 



5:00 Tests for Independence in Contingency 
tables. 



Tests of interaction across contingency 
tables and associated post hoc proce- 
dures. 



The Sign test and Stuart’s tests for 
trend. 



1:30 - 3:00 



Spearman’s • o 
for matchec 



the wilcoxon test 
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3 "30 - 


5-00 


ftew a ill 9 g cor 1 i c ie n t c f oncordance 

u./i ci the Fn dm an 


8: 30 - 


JLAv \j\_t 


. ' hoc pie . "Go * bi*e friedman 


10 ; 30 - 


1 it : uG 


Cochran’s Q cost anu associated post 
hoc procedures. 


1:30 - 


3:00 


The McNernar test as a special case of 
the Q test. 


3:30 - 


5:00 


Rank analog to Hotelling’s T 2 and cor- 
responding post hoc procedures. 



5 



<3:30 - 10; 00 

10; 30 - 12; 00 
1:30 - 3 "00 

3:30 - 5:00 



V/ilcoxon test (Mann-Uhi 
cation of the Wilcoxon 



tney ) and appli 
test to block 



designs • 



Test on aligned observations. 

Normal scores test for the two -sample 
problem. 

Kruskal-V/allis test and associated post 
hoc procedures. 



rJTIjPY AND HEFijlMIifk UVTr«:i?IAI,S 



Each staff member prepared five detailed lectures. These lectures were 
presented and given to the students. They were carefully followed and used 
as the lecture proceeded. Participants could refer to the printed materials as 
the instructor lectured and, therefore, did not have to write notes and copy from 
the blackboard to comprehend and follow the train of thought being oppressed by 
the instructor. 1 .. Since it was assumed that the blackboard facilities would not 
be adequate, the lectures were prepared in detail. Copies of the lectures are 
attached to this report. 



Prior to the Presession, letters were written to the authors of pertinent 
journal articles for permission to xerox the articles for distribution to the 
participants. Every author gave this permission. The articles were, therefore, 
xeroxed and presented to the students. 

Each student purchased an inexpensive but excellent text book on nonpa rametric 
methods, Distribution-Free Statistical Tests by James V. Bradley. The manuscript 
was prepared by the Aerospace Medical Division, Wright- Patterson Air Force Base, 
Ohio, and is available to the public via the United States Department of Commerce. 
The text cost each student $3.25: three dollars for the text and 25q for the mailing 
of the text to Chicago from Berkeley. 
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EVALUATION^ MATERIAL S 



Each student was given three multiple choice tests containing very 
difficult items. The tests are included with this report as an appendix. 
The statistics for the three tests are as follows: 



Test 


Average No. Correct 


Standard Deviation 


One 


12.2 


5.6 


Two 


13.1 


4.1 


Three 


22.6 


.9 



Examinations one and two were given on the third and fourth day of the 
Presession. Test three was given to the participants on the last day. They 
were asked to mail the completed answer sheets to the Director. Only ten 
completed answer sheets were received. 

The low scores on the tests are not unusual nor were they unexpected by 
the staff. fhe questions were deliberately written at a difficult level because 
it was incorrectly assumed that the participants would be better prepared than 
they actually were. While knowledge of the binomial and hypergeometric distrib- 
utions is essential to the use of many nonparametric methods, it turned out that 
most participants had only superficial knowledge of these important statistical 
distributions. Another weakness noted by the staff is the relatively poor 
training that students in schools of education obtain concerning the importance 
of confidence intervals. It seems that most instructors of statistics over- 
emphasize the testing of "null hypotheses" and ignore the more important 
procedures associated with confidence intervals and their use for making 
statistical inferences. Since the Presession emphasized the use of simultaneous 
multiple confidence interval procedures, almost all of the material presented 
to the participants was "new". 



EVALUATION BY STAFF 

At the end of the Presession, an item questionnaire was given to each of the 
five staff members for completion. The questionnaire dealt with matters concerning 
the physical environment, the scheduling, and the organization of the Presession. 

It allowed comments to be made by the staff regarding their perception of the 
participants. A list of the items and a tally of the five staff members* responses 
are given. Staff members "Commendable" responses are tallied in column "C", 
their "Satisfactory" responses in column "S", and their "Unsatisfactory responses 



1. Environmental conditions 
a* Classroom spaces 
b* Work spaces 
c. Living quarters 

d* Teaching equipment, aids (chalk boards } . 

public address system, etc.) 
e; Resource material, library 
f# Eating facilities 



in column "U" . 



U 



1_ 

1 



3 2 

-j ; 

i 

— i— -JaL_ 
Jt 

- Ik - 
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— 


1 




X 

~ 


l 


— 








2 


_X_ 




mmmmmmmrn 


2 


IE 



5 



1 


-A- 




1 


f± . . 




2 


2 


1 


1 


Jt- 


_ 


— 


_2_, 






1 


Jt_ 


T 


A 




~ 


5 






h- 




_ 


-JL 


~ 




2 




2 


JL 




JL 


2 






JL 




— 


2 


X 


2 


2 


1 


1 


k 




2 






JL 


2 





2 . 



3 . 



!+• 



5 . 



Participants 

a. Appropriateness of academic backgrounds 

b. Sufficiency of research experience 

c. Willingness to work 

d. Intellectual curiosity 

e. Concern for applicability of techniques 

f. Aspiration 

g. Immediate preparation for Presession 
Organization 

a. Adequacy of notice to applicants 

b. Sufficiency of preplanning 

c. Smoothness' of operation 

d. Adaptability to obstacles and feedback 

e. Sensitivity to grievances 

f* Adequacy of financial support 

Schedule 

a. Appropriateness of five days for the job 

b. Time spent efficiently 

c. Events sequenced appropriately 

d. Punctuality 

e. Balance between formal, informal affairs 

f. Quantity of discussions 

g. Quality of discussions 

h. Quality of formal presentations 

i. Unobtrusiveness of evaluation efforts 

j. Methods of evaluation 

Outcomes 

a. Intended content was actually taught 

b. Increase in participant understanding 

c. Improvement in attitude toward research 

d. Personal associations initiated 



For the most part, the staff agreed on most issues. They were not in agree- 
ment on adaptability to obstacles and feedback, on the quantity of discussion, 
and on the intended content actually taught. The staff thought that the teaching 
equipment, the appropriateness of academic backgrounds, immediate preparation 
on the part of the participants for the Presession, the appropriateness of a five- 
day Presession, and the methods used for evaluation were unsatisfactory. If this 
Presession is given again, these characteristics will be given extra attention. 

EVALUATION BY PARTICIPANTS 

The participants took full advantage of the opportunity offered them to comment 
on the operation of the Presession. A summary of their responses, together with 
particularly germain comments, follows: 

1A. To what extent did the relative availability or unavailability of books and 
journals interfere with or promote your attempts to master the content of this 
session? 




Response 



Frequency- 



Percent 



None 


35 


83 


Little 


4 


10 


Some 


3 


7 




42 


100 



Participants were given detailed handouts and therefore did not have to 
refer very frequently to textbooks and research articles. Also, participants 
were told to bring copies of statistical tables which are usually found in 
most elementary statistics texts. Thus, lack of reference materials was not 
a real problem for the participants. 



IB. To what extent did reproduced materials given to you by the staff 
improve matters? 



Response 


Frequency 


Percent 


Helpful 


8 


19 


Very Helpful 


34 


81 




42 


100 



4.1 staff was 9 uite pleased with the reception the participants gave to 

the handouts. In this case, one can be sure that they have not become throw-aways 
w s reported that the handouts made all the difference, were tremendously help- 

“ ’ rfi? e f e ^ S a e p tlal t0 u he course ’ lt was also noted by some that the handouts 
were well fitted for use when the participants returned to their home base. 



2A. Did you feel t^at you lacked a "place to work " 
small groups? ’ 









Response 


Frequency 


Perc 


Yes 


9 


21 


No 


31 


73 


No Comment 


2 


5 




42 


100 



Since no homework was assigned, a working or discussion place was not 
really needed. 

2B . Was your room satisfactory? 



Response Frequency Percent 

Yes 33 79 

No 7 17 

No Comment 2 5 

42 100 



Some of the rooms were not in the best of condition, 
some rooms were quite fine and excellent. 



On the other hand. 
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3A. Which features of the meeting rooms were inadequate or not 
conducive to learning? 



Response Frequency 



Percent 



Blackboards 29 
Size 17 
Light 2 
Sound 1 
Air 7 
Furnishings 6 
Lack of Overhead 

Projector 3 
Miscellaneous 3 



68 



43 

25 

3 
1 

10 

9 

4 
_4 

100 



The blackboard facilities were atrocious. They were not conducive to 
the staffs’ needs or presentations. They were wobbly, difficult to write 
upon, and next to impossible to read. More attention must be given to these 
facilities when selecting future AERA Presession sites. According to some 
participants , the chalkboards were too small. The crowded seating conditions 
and the lack of work space were frequently complained about. Simple lack of 
elbow room was a common complaint. 



3B. Which features were especially facilitative in the same regard? 



Response 



Frequency Percent 



Convenience 

Light 

Sound 

Air 

Furnishings 
No Comment 




5 

3 

12 

3 

5 

72 

100 



The room was large, generally airy, and bright. Most of the participants 
could hear the speakers quite well. The majority welcomed the water pitchers 
because of the stuffy afternoons. Two mentioned the convenience of having the 
meetings where they were residing. 



4A. Was five days too long a period to leave your work at home for the 
purpose of attending this session? 



Response 



Frequency 



Percent 



Yes 

No 



7 

35 

42 



17 

-SI 

100 



o 
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Five days was clearly not too long The staff felt that two days 
should he spent on review of classical statistics with an expansion of the 
probability notions associated with the binomial, hyper geometric, and 
uniform distributions The participants were ill-prepared to handle con- 
loence intervals and multiple post hoc procedures. The nonparametric 
material could have been extended and presented at a slower pace and with 
a greater emphasis on experimental design A ten-day presession with a 
well-organized laboratory section would have made the presentation more 
valuable for most students and a bit more relaxing for the staff Both 
students and staff worked hard. 



4B. Was five days too short a period in which to learn much of the 
content of this session? 



Response 



Frequency Percent 



Yes 

No 



35 83 

_T7 

42 100 



Five days was too short. One 
the staff was the necessity to stifle 



thing which was quite disheartening to 
and cut short any tangent discussions. 



Without question, these extraneous discussions are useful. On the last 
day, the Director did become involved in a discussion of experimental 
design related to his own research. The response and reaction of the 
clUoS to these kinds of discussions was always positive and reinforcing. 
As one. student reported, more linkage of methods to design was needed. 
Next time, if there is a next time, these discussions will be encouraged: 
but for them, more classroom time is needed. 



5A. Were you allowed enough time in which to pursue activities of 
your own choosing? 



Response 


Fre quency 


Precent 


Yes 


28 


67 


No 


13 


31 


No Comment 


1 


2 




42 


100 



In the evening the participants were free to attend plays, concerts 
movies, sports events, cocktail bars, dinner, and. even study if they so 
esired. The staff felt that 8:30 a.m. to 3:00 p.m. on nonparametric methods 
was a lot to ask. of any student. Therefore, an attempt was made to let the 
participants unwind after lectures. However, the individuals who responded 
No to this item felt that much of their free time was occupied by keeping 
up their classwork and reviewing the materials presented during the day. 



0 
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5B. Would you have preferred not to meet in the evening after dinner? 



Response 


Frequency 


Percent 


Yes 


12 


29 


No 


10 


24 


No Comment 


20 


48 




42 


100 



Since there were no evening meetings, this question was not directly 
applicable. Perhaps the "yes" answers referred to general feelings about 
evening meetings after all day sessions. 

5C. Would you have preferred more or fewer meetings per day than 
there actually were? Or was the number of meetings per day agreeable to you? 



Response 


Frequency 


Percent 


Fewer 


5 


12 


Enough 


36 


86 


More 


1 


2 




42 


100 



The staff agrees with the participants. Four meetings a day are 
enough. However, one meeting devoted to laboratory work or group discussions 
would still be desirable. 



6A. Were the individual lectures too long to sit and listen or 
take notes? 



Response 



Frequency 



Percent 



Yes 

No 



14 

28 

42 



33 

67 

100 



Some of the lectures were definitely too long, or seemed to be, since 
the material was sometimes very heavy and abstract. As one participant 
reported, "Too much material is tiring." 

6B. Were the lectures scheduled in an appropriate sequence? 



Response 



Frequency Percent 



Yes 40 95 

No 2 £ 

42 100 



Considerable time was spent in organizing the material so the new 
ideas and new tests could be built on foundations laid earlier. The staff 
seemed to be quite successful in using this spiralling type of presentation. 
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7. Did you have sufficient opportunities to interact with other 
participants? 



Response 



Frequency Percent 



Yes 

No 



25 

17 



60 

AQ 



100 



The staff felt that they did not have a chance to interact with 
enough of the participants. Perhaps part of the problem is related to the 
length of the Presession. It takes more than five days to be able to communi- 
cate and discuss educational research problems on a common level of understanding. 
It . takes time for people to get to know one another and for individual person- 
ality characteristics to emerge. Verbal interaction or communication between 
people who are essentially strangers is difficult. Perhaps a cocktail party on 
the night before the Presession or the group discussion periods which were 
planned but abandoned would have helped to improve intercommunication and 
interaction. Without doubt there should be more opportunities for this kind of 
communication. 

BA. Were the instructors too inaccessible or unapproachable so that 
you did not get the individual attention that you desired? 

Response Frequency Percent 

No 41 98 

No Comment l 2 



100 



The staff is quite flattered, but wish they could have done more. 
8B. Was it helpful to have graduate student assistants present? 



Response 



Frequency 



Percent 



Yes 

No 

No Comment 




33 

4 



79 

10 

12 

100 
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Without the graduate assistance, the Presession would have fallen 
on its face. The behind the scenes activities that they performed for the 
teaching staff were invaluable. If the discussion sections could have been 
held, there is no doubt that the work of the graduate assistants would have 
been first rate and of immense value to the participants. 

9. Did the attempts to evaluate your progress and reactions during 
the session (and at this moment) interfere with your work here? 



Response 


Frequency 


Percent 


Yes 


11 


26 


No 


30 


71 


No Comment 


1 _ 


2 




42 


100 



The evaluation was not very satisfactory. The tests were too difficult 
for the background possessed by the participants and the tests should never 
have been given. They were only given because of this report. They did create 
anxiety and in that sense were harmful. 

10. In general, was the Pre session well organized? 



Response 


Frequency 


Percent 


Very Well 


21 


50 


Well 


19 


45 


No Comment 


2 


-1- 




42 


100 



When the tremendous amount of work that was performed prior to the 
Presession on the part of the staff is noted, it is very gratifying to learn 
that the participants were appreciative. Unfortunately, a couple of the 
participants felt that the Presession was a little too well organized or 
over-organized . 

11A. Did the content of the lectures and readings presuppose far more 
previous training than you had? 

Response Frequency Percent 

Yes 28 67 

No 14 33 

42 100 

This was unfortunate; yet the staff continually lowered it's horizon 
as the Presession progressed. As one student reported, "I still wonder if 
you were following dissonance theory, throwing so much at us on the first 
day." But, fortunately, several participants felt that the instructors 
were taking into account the level of the participants and were restructuring 
as they went along and, indeed, this was true. 
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11B. Should less training in these areas or more have been pre- 
supposed? 



Response Frequency Percent 

More 5 12 

Neither 13 31 

Less 24 57 

42 100 



Next time, less training will be assumed. 

12. To what extent was the content of the lectures and readings 
relevant to what you hoped to accomplish during the session? 



Response 


Frequency 


Percent 


Some 


12 


29 


Much 


25 


60 


No Comment 




12 




42 


100 



The staff felt through the questions asked that some participants 
were really there to learn experimental design. Next time, advanced 
preparation for these students will be made. Without doubt, inclusion of 
experimental design material would have improved the Presession considerably. 
Even so, one participant reported, "This five-day session has opened up new 
vistas for me." 

13A. Were the lecturers stimulating and interesting? 

Response Frequency Percent 

Yes, Usually, 

or Somewhat 26 62 

Very 16 38 

42 ion 

According to one participant, "The staff is to be commended for their 
excellence." Again, the staff thanks the participants. 

13B. Were the lecturers competent to speak on the subject assigned 

them? 

Response Frequency Percent 

Yes 42 100 

No 0_ 0_ 

42 100 

13C. Were the lecturers well prepared? 
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. o 

me 



Response 


Frequency 


Percent 


Yes 


42 


100 


No 


0 


0 




42 


100 


14. Were you disappointed in any way with 


the group of participants? 


Response 


Frequency 


Percent 


Yes 


5 


16 


No 


67 


84 




42 


100 


The participants were 


eager, receptive, and interested. They were 


a pleasure to have as students. 




15. If you had it to 


do over again 


would you apply for this 


Presession which you have just completed? 




Response 


Frequency 


Percent 


Yes 


40 


95 


No 


2 


P 




42 


100 


16. If a presession such as this is 


held again would you recommend 


to others like you that they 


attend? 




Response 


Frequency 


Percent 


Yes 


42 


100 


No 


0 


0 




42 


100 



17. Do you anticipate maintaining some sort of contact with at least 
one of the Presession staff? 



Response 



Frequency 



Percent 



Yet 

No 



26 

JLL 

40 



65 

100 



The staff also hopes to do the same. 



16. Do you feel that AERA. is making an important contribution to 
education by sponsoring presessions such as this one? 



Response 



Frequency Percent 



Yes 

No 



42 100 

0 0 

42 100 



* 
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Without doubt the Presession was of immense value. The new ideas 
presented to . the participants are going to improve their work and are 
certainly going to have an impact on education. 



19. Do you feel that anything has happened during these five days 
to make it more likely that you will leave your present position of 
employment? 

Response Frequency Percent 

5 12 

37 88 

42 100 

20. Is it likely that you will collaborate in research with someone 
else attending this Presession (other than those you already were likely to 
collaborate with)? 



Yes 

No 



Response 



Frequency Percent 



Yes 

No 



14 33 

28 67 

42 100 



21. Do you think that the staff should feel that it has accomplished 
its objectives during this five-day Presession? 



Response 


Frequency 


Percent 


Yes 


35 


83 


No 


3 


7 


Equivocal (due to 
question wording) 


L. 


10 




42 


100 



The staff does not think it satisfied all of its objectives which 
were soon seen to be unrealistic . However, the staff is convinced that it 
has opened new approaches to research problems to the participants. Further- 
more, the staff is convinced that the participants will ask new and different 
kinds of questions because of the exposure to the new ideas presented in the 
Presession 



SUMMARY 



The Presession on Nonparametric Methods and Associated Post Hoc 
Procedures was held at the Sheraton Chicago Hotel on the five days just 
prior to the 1968 AERA Convention in Chicago. While most of the partici- 
pants were unprepared for the course planned by the instructional staff, 
the number of new ideas and the intellectual growth that has taken place in 
the participants clearly shows that the Presession has been successful. While 
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it did not satisfy all of its originally stated objectives, it did open 
new vistas and new ways for looking at data collected in educational 
settings. The value in conducting this Presession will be measured in 
the payoffs to education. Both the staff and the participants expect this 
payoff to he large. 
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Examination One. 

i. Compared to classical tests of* hypothesis, distribution 
free tests ares 



2. 



3. 



a. easy to understand since they are generally 

derived from simple applications of proba- 
bility formulas. ^ 

b. easy to conduct since the arithmetic required 

is generally very simple, often involving 
simple counting. 

c* are often superior to or equal in efficiency 

to the classical test when the classical test 
is not appropriate, 
d. all of the above, 
e • none of the above • 



If X is a binomially distributed random variable such 
that on any one trial P(A) ~ -J, what is the proba- 
bility of exactly two A 9 s in three trials? 



a. 

b. 

c. 

d. 

e 



17* 

2/a 

3/8 

5/8 

a/a 



The variance of the contrast 4=r (pi f p 0 

estimated by; ^ 1 ^2 ^3 > 

a - ii h.P t rih \ f hi 

^ i. .^K -J 0 



b * P [A >- PPi - ~ h ii 

A A 



A 



/ \ 



X p 

3 \y 






\ 



l_%> -t- i jfJ i l i 



+■ 






' v '- ; r— -T 



<r v t 

'A a 

(P c 7 , M 

/A-. 

M 

^ 1 



“• ^ * jijf. „ A* J * £vj. 



e. None of the above. 






is 



4« If Z is a normally distributed random variable with ex- 
pected value of zero and variance of one, then Z 2 has 
a sampling distribution which is 



a. normal. 

b. hypergeometric 

c. Chi-square. 

d. binomial. 

e. none of the above. 



5 . 



The test statistic for the classical test of 

Hq“ q} ’ r ci ^ has a nu ll distribution which is 

a, normal o 



b. 

c. 

d. 

e. 



. . ■ ~> 

Ny | rv ■ 2_ * — 

c 

^|-l> KJ Z - 1 
hypergeometric • 



xi, 



w,-1 



6 . Which of the following are valid combinations of the 

parameters for the application of the Chi-square analog 
to Scheffe v s theorem? Mark as many as applicable. 






a. p x - p 3 



b. 



c. 



+- /h 2 




Pi ‘ P 2 4- P 3 



d. X <t>, - {>•*_ - +■ 2 

e ' ~ 3 T |> d t #> w 4- 3^>v- 

7. If the Y 9 s are independent, according to the Central 
Limit Theorem, the sampling distribution of 
Tn Yu- Y 2 4 - 134 - ... +- Y n , where E(y ± ) = and 

Var (Y^) - (3^~ , is approximately normal with var- 

iance given bys 

a * $7Z 

b. C^Y N - ^ ) 

hi \ v~ I ) 

c. M c}"*" 

d. Si! 

r-J 

e * L d 1 

N 'Tvo 



o 

ERLC 



1 u 



S* From a finite population of size 50 with variance equal 
to 36 , a random sample of size 7 is selected. The 
variance of the sampling distribution for means of 
samples of size 7 is given by” 



a. 


Ql V 


H, 








Ki 


So 






b. 


o L - 


L 










1 


- 




c.‘ 


d l { 


/V -*/K> 


\ 3 4 7 


iO- 7 \ 






Sj — i 4 


/ - 7 V 


- 1 ) 


d. 


di 1 


V-( - 


U Ik i 


c 

1 




Kl V 


/V-/K. 


' ‘ ro * 


< so - ) ) 


e • 


none 


of the 


above . 





9. Nonparametric tests can legitimately be applied to a much 
larger set of situations than parametric tests because 

a. they require fewer assumptions, 

b. they require less elaborate assumptions. 

c. they have greater versatility, 

c. all of the above 

e . none of the above . 



10. A distribution-free test 

a. makes no assumptions about the pred se form 

of the sampled population. 

b. makes no assumptions about the precise value 

of the parameters of a sampled population. 

c. requires that the sampled population be 

symmetrical 

d. does not substitute other quantities for the 

observed values. 

e. none of the above. 

11. The mean square contingency coeffient denoted by 

is a number ' /w 

a. between 0 and 1. 

b. greater than or equal to 0 

' c. less than or equal: to 1. 

d. unbounded in both directions. 

e. none of the above .1 
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12 . 



When the hypothesis of the Irwin-Fisher test is 
the resulting test statistic has 



true. 



a. * at distribution* 

b. a binomial distribution. 

c. a Chi-square distribution. 

d. a normal distribution. 

e. a hypergeometric distribution. 

13 • The assumptions for the Irwin-Fisher test are 

a. independent random samples from identical 

continuous distributions. 

b. match paired observations from a binomial 

distribution. 

c. independent random samples from binomial 

distributions. 

d. independent random samples from distributions 

with identical variances. 

e. correlated samples from hypergeometric dis- 

tributions. 

14* The Chi-square test of independence is 

a. derived from normal probability theory 

b. used to test the hypothesis of equality of 

medians. 

c. a test which assumes equality of variance, 
a. a one sample test. 

e. none of the above. 

15* For an R by C contingency table the number of degrees of 
freedom is given by 

R(C - 1) 
c (R - 1) 

(R - 1) (C — 1) 

RC 

None of the above. 

sample approximation to the Irwin-Fisher Exact 



3 • 

b. 

C o 

d. 

e . 

16. The large 
test 



a. 

b. 

c. 

d. 

e. 



assumes n-. n n 0 



Y, - £(y t ) 

utilizes ~z_ ^ - 1 ■ - 

requires a correction for continutity. 
is a one sample test, 
none of the above. 



The Irwin-Fisher test is the nonparametric analog to 

a. The Welch-Aspin test, 

b. the two sample t-test, 

c. the K sample F-test. 

d. all of the above, 

e. none of the above. 

The Irwin-Fisher test is used to test 



a, p ± - p 2 
b • "2 




M 1 ~ 
Pi = 



O , 



M 2 
P2 — 



C> 



z. 



• Pk 



The principal reason for using the arc sine transforma- 
tion — 2 Ojic4i,vu ^ n place of the proportion p y 



k. 



in the statistic Oo an d in contrasts is to 



a, test the hypothesis of equality of the popu- 

lations in terms of more easily interpreted 
parameters, 

b, simplify computations, 

Co increase the degrees of freedom in the com- 
putation of 

d, guarantee that the null distribution of L)1 

is approximately Chi-square. ° 

e. Increase the power of the test 

Which is not an assumption needed to insure valid use 
of the Chi-square approximation to the Irwin-Fisher test 

a. the expected frequencies should be greater than 

or equal to 5« 

b. Independence between observations. 

c. The underlying distribution of the variable 

is normal. 

d. The probability of the qualitative variable 

is constant over subjects. 

3 * none of the above . 
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21. If in a 2 x 2 . contingency table it is found that (£> is 

not statistically different from zero, then it is 
known thata type one error has been committed or that 

a • o 

b. A ~ I 

C. X- I i 

d * P( ^ B d ) - P( 4 j P( (3 1 \ A ^J) 

e * PC U f3. > - p ( a .O PC 13, ) 

22. The phi coefficient 

a. is derived from Pearson ? s Product Moment 

Correlation Coefficient. 

b. assumes that the two variables are normally 

distributed. 

c. assumes that one variable is normally dis- 
tributed and the other is binomial. 

d. is not a valid measure of association. 

e. none of the above. 

23. An investigator who uses a Chi-square test for homo- 

geneity to test Hq; P] - P 2 r r py in a 2 x 7 

table rejects his hypothesis. He decides to use the 
Chi-square analog to Scheffers theorem to examine 
differences among the p,. The value of \j to be 

used in the confidence interval would have how many 
degrees of free dome? 

a. 1 

b. 2 

c . 6 

d. 7 

e. indeterminate from the information given. 

24. How many confidence intervals may the investigator of 

question 23 set up if he uses the Chi-square analog 
and wants to hold the overall probability of a type 
one error to .05? 

a. 1 

b . 6 

c. 7 

d. a limited but unspecified number. 

e. an unlimited number. 
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Examination Two 0 



1. An investigator is interested in the auditory discrimi- 
nation of 50 matched-pairs of children* In computing 
the differences, Xj - Y-j, he finds 10 pairs of tied 
observations Xj « y^ which he discards* The Wilcoxon 
matched pairs test is computed on the remaining ob- 
servations# His procedure is 

a# desirable because it discards observations 
which give no information* 
b# undesirable because it reduced the power of 
the test* 

c. desirable because it permits the use of 
exact tables* 

d» undesirable because it increases the power 
of the test spuriously, 
e* either desirable or undesirable* 

2# The Wilcoxon matched-pairs test can be used to test the 

hypothesis Hg : ■- 50 as well as Hq- M ~ 50 if the 

population distribution of scores is 



a* negatively skewed* 
b# normal. 

c* positively skewed, 
d. all of the above* 
e* none of the above. 



3* The Wilcoxon matched-pairs test for related observations 
is the nonparametric analog of 



a. F-test, 

b. t-test, 



d* 

S o 



F ~ MSB/MSW 
t - (X - Y)/S p 1/ 



N v, 



Z-test, Z » (I - Tj/Uli. + Q y- 



In 






M s 



y 



A c 7 - 



c z 



Welch-Aspin, t v -•? (X - Y )/| : AJL. £ X, 

; w * N •/ 

none of the above. 



A* An administrator who is interested in the possible effect 

of school district consolidation on pupil-teacher ratios 
compares the median pupil-teacher ratios for 60 pres- 
ently consolidated districts before and after consoli- 
dation. He believes that the underlying distribution 
of pre-post consolidation differences is extremely 
positively skewed. Indicate the 



most and least appropriate test statistics respectively 
for his hypothesis. 

a. Wilcoxon matched-pairs test, Randomization 

t — t G S t 

b. Sign test, Wilcoxon matched-pairs test 

c. Randomization t-test, Sign test 

d. Wilcoxon matched-pairs test, Sign test 

e. Randomization t— test, Wilcoxon ‘matched- 

pairs test 



The hypothesis that scores increase monotonically over 
time can be tested by the Cox-Stuart tests for trend. 
The hypothesis can also be tested bv 



a. Fisher randomization t-test 

b. Wilcoxon matched-pairs test for a simgle 

sample. 

c. Wilcoxon matched-pairs test for related 

samples » 

d. Significance test for Spearman rho 

e. None of the above. 

The median test for 2 independent samples is 



a. more powerful than the t-test 

b. derived from the sign test 

c. requires equal sample size 

d. symmetrical about the median 

e. none of the above 



For the interaction 



test Hq" w r- o where 

tested against H-^ H Q is false 5 



a. The underlying variables must have equal 

standard deviations. 

b. The underlying variables are correlated. 

c. The appropriate test statistic has a distribu- 

tion that is approximately Chi-square with 
K - 1 degrees of freedom. 

d. Has an associated post hoc comparison method 

that is based upon the assumption that all 
<£> ^ are equal and all Pz^ are equal. 

e. Is in reality only a K sample problem. 



The Sign test for the median difference 

a. Cannot be used if each pair is selected from 

a different population. 

b. Cannot be used if each pair is measured in 

different units. 

c. Cannot be used if one population is symmetrical 

and the other is skewed. 

d. None of the above, 
e* All of the above. 
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9#~ The test statistic 2L~ ( p> { 



is the large sample form of 

a. The sign test 

b. The Chi-square test 

c. The Wilcoxon matched pair test 

d. * The median test 

e. The Irwin-Fisher test 

10* To carry out the median test on two samples, one must 

a. find the median of each sample 

b. determine the combined median 

c. take the difference between corresponding scores 

d. rank the scores 

e 0 none of the above 1 

11. When using the Sign test as a substitute for the clas- 

sical match pair t— test one must assumes 

a. Pairs are infependent and P( positive deviation) 

equals one-half. . 

b. Pairs are independent and variance is known. 

c. Distribution of deviations is symmetric and 

pairs are independent. 

d. the population is uniform with unknown variance 

e. that the assigned ranks are equally likely 

and that there are no tied values 

12. The Wilcoxon one-sample match pair test is 

a. more powerful than the Sign test 

b. less powerful than the Sign test 

‘ c. cannot be used as a su ^stitute for the two 
smaple t-test 

d. cannot be approximated by the Normal Dis- 

tribution 

e, a special case of the Irwin-Fisher test 

13. The Irwin-Fisher test is based upon 

a. ranking the plus and minus samples resulting ftom 

performing the Sign Test. 

b. the binomial distribution. 

c. sampling with replacement. 

d. all of the above. 

e. none of the above. 










IMP 
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14« If the linear contrast following the Friedman Test is 
significantly different from zero and positive 

a. The median of the population from which the 
sample came also has a linear component • 

* b. The expected values, of the populations from 

which the samples came also have a linear 
component. 

c. The variances of tljie populations from which 

the samples came increase as k increases. 

d. The expected average ranks have a linear form. 

e. The centers of the | distributions increase 

as k increases. 

15* The 2 x 2 x K contingency tab^.e problem 

a. has no simple 2x2 contingency table counter- 

part .* 

b. can only be performed if all estimated ex- 

pected frequencies exceed 5» 

c. can be performed if not more than 20 % of the 

cells have estimated frequencies that 
exceed 5* 

dp is tested against a simple alternative as 
as apposed to an omnibus alternative. 

3« is based upon a Chi-square statistic of the 
Karl Pearson Form ^ ( q- £ ) z 

16. In a study involving men and women, who were later 

classified as above or below average intelligence and 
were also rated as to their ability to complete a 5 
day course on nonparametric statistics it was found 

that u 0 ' - >2.^24, 

It can be concluded that 

a. for the women, expected performance is unre- 

lated to intelligence. 

b. for the men, there exists a positive relation- 

ship between expected performance and in- 
tellegence. 

c. the confidence interval for — X will 

lie completely above zero. 

d. all of the above. 

e. none of the above. 
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17. 



Sign Test probabilities can always be computed from 



a. the hypergeometric formula, 
b* the binomial formula. 

c. the normal curve f •. ula. 

d. none of the above, 

e. all of the above. 



1$. Compared to the classical t-t^st based on 50 observa- 
tions, the Sign Test with almost the same power must 
use approximately 

a. 20 observations 

b. 32 observations j 

c. 64 observations 

d. SO observations 

e. 200 observations j 



19. The Match Pair Median test performed as a Sign Test can 
also be. considered a test of 

a. the underlying distributions are skewed 

b. the samples are not random 

c. the observations are not independent 

d. all of the above 

e. none of the above i 



20. The Sign Test for the median difference has its greatest 

power when zero differences are discarded.- Even though 
the power is greatest under this condition, it is not 
recommended because 

a. a zero difference being halfway between a 

plus or a minus is really supportive of 
the hypothesis. 

b. the observations in the sample are corre- 

lated. 

c. the resulting test statistic is difficult 

to compute • 

d. the probability of a type one error is 

increased. 

e. the power is not increased to justify the 

discarding of t9 embarassing i? data. 

21. If one adopts a conservative point of view when conducting 

a test of hypothesis, the appropriate procedure to 
follow when handling tied values in a nonparametric 
test is to: 

a, discard the tied values and reduce the sample 

sample size accordingly. 

b. randomly break the jties for each tied value by 

flipping an unbiased coin. 
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c. divide the tied values equally between the two 

adjacent categories and thereby maintain the 
original sample size. 

d. divide the tied values so that the resulting 

outcomes are most compatible with the alter- 
native hypothesis. 

e. divide the tied values so that the resulting 

outcomes are most compatible with the hypo- 
thesis being tested. 

22. The confidence interval for the median is based on 

a. the hypergeometric distribution. 

b. the normal distribution. 

c. the Chi-square distribution. 

d. the Binomial distribution. 

e. ' none of the above. 

23* If the Sign test for the medihn difference is tested 
against the one sided alternative that Method A is 
superior to Method E, and if the hypothesis is rejected, 
one can conclude in the distribution of original in- 
terest; 

i 

a. that the expected values of the two populations 

are different. i 

b. that large positiye deviations are counter- 

balanced by large negative deviations with 
corresponding statements applying to small 
deviations. 

o. that more than one half of the positive 
deviations are greater than zero. 

d. that the elements lof one population are on 

the average greater than the elements of 
the other population. 

e. that the variances! of the two populations 

are different. j 

24« If the hypothesis that M =. 32 has been rejected by the 
Sign test, then the confidence interval for the un- 
known median 

j 

a. is narrower than that of the expected value. 

b. is certain to contain 32 . 

c. might contain 32 . 

d. is certain not to jcontain 32 . 

e. none of the above. 1 

! 

25* The Sign test can be used to test 

a. that the median of a population is equal to 
a specified value. 
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b. that the median difference for a pre and 
post test design is equal to zero* 
c* that the expected difference in a match pair 
design is zero* 

- d* none of the above. 

e* all of the above* 

EXAMINATION THREE 

1 

1* For the Resenthal and Ferguson post hoc procedure for 

Friedman Contrasts, - o_, J2., -t-a z C * + i . ■ V C- k E* 

the w Scheffe f? coefficient is given bys 

c * i~r 

V /W.- « + , *-',**' '< 

d * - 

e * - 1 

2* The Wilcoxon matched pairs test is a 

a* test for independence* 
b* one sample test* ? 

c* substitute for the 'X ^ test of homogeneity, 
do substitute for the F-test. 
e* none of the above 

3. The decision rule for the Sign test has been defined as 

reject H 0 if X £. (0, 1, 5), the probability of falsely 
rejecting Hq when it is true, is given by 

a. 1/32 

b. 2/32 

c. 5/32 

d. 7/32 

e. 12/32 

4* Stuart’s tests for trend V ^3 are special cases of 

a. the normal curve test. 

b. the Wilcoxon test. 
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c. the Sign test. 

d. the Irwiri-Fisher Exact test. 

e. the t-testo 

5. The Kruskall-Wallis test for K groups is the nonparametric 
analog to 



a. the Wilcoxon test. 

b. The F-test for repeated measures. 

c. the VIelch-Aspin test. 

d. the Chi-square test. 

e. the One-way Analysis of Variance. 

6. The exact hypothesis tested by the Kruskal-Wallis test is 

a* K universes have identical distributions. 

b. K universes have equal medians. 

c. K universes have equal expected values. 

d. Pi * P2 « ••• “ PK 

e. K correlated populations are equal. 

7. For the Kruskal-tJallis test, tied scores 

a. tend to make H a conservative test. 

b. make the test invalid. 

c. cause the power to increase. 

d. make the confidence intervals narrower than 

they really are. 

e. are dropped for the analysis. 

8m The test of HqS ~ ^ ° -Yk against H]_s Hq is false, 

where Iv-j c pu ^ ^ 

assumes 

a. K match paired samples from binomial distri- 

butions. 

b. K independent random samples from multinomial 

distributions. 

c. 2K independent random samples from binomial 

distributions. 

d. 2K correlated samples from hypergeometric 

distributions. 

e. K large independent random samples from K 

bivariate qualitative variable universes. 

9. For the Friedman Test post hoc procedures are based on 

contrasts of the form 4-CL^ u ^/<L 

The variance of these contrasts is given by 
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10 . 



3 * 



k ( a±<) 
n ^ 




_ Z 




e ' b^zJ oJjt, ! 

i 2- ^ Vk> 

If one is interested in q specific contrasts when 
q< (K - 1), I 

a. it is necessary to first test the K-sample 

omnibus hypothesis of equality. 

b. one can determine the q confidence intervals 

without first computing the test statistic. 

c. one should increase the sample size for the 

population of interest. 

d. one should decrease 1 the value of c< associated 

with the K sample omnibus test. 

e. one should determine each confidence interval 

with significance level set at • 



11. The Friedman test is an extension of the 



a. Median test. 

b. tyilcoxon Matched-pairs test. 

c. Chi-square test. 

d. Sign test. 

e. none of the above. 

12. The degrees of freedom for the Friedman test equal 

a. the number of blocks minus 1. 

b. the number of conditions minus 1. 

c. the number of blocks times the number of 

conditions. 

d. the number of blocks minus 1 times the number 

of conditions minus 1. 

e. none of the above. 

13. For the Friedman test, the hypothesis under test states 

a. the medians are the same for all conditions. 

b. the means are equal over conditions. 

c. the expected average ranks are equal over all 

conditions. 
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d. the proportion of successes in each condition 

are equal. 

e. none of the above. 



14. Kendall’s Coefficient of Concordance 



a. 

b. 

c. 

d. 
© • 



lies between -1 and 1. 

shows a measure of the association for the 
Chi-square test of independence, 
is a linear function of the Wilcoxon test 
statistic. 

is derived from Pearson’s Product Moment 
Correlation Coefficient, 
none of the above. 



15. To compute the Friedman test 



a. 

b. 

c . 

d. 
e • 



rank each block separately. 

rank ever all conditions. 

sum the scores over each condition. 

rank the scores in each condition separately. 

none of the above. 



16. * The asymptotic efficiency of the Wilcoxon test to the 
Mann-Whitney test is 



a. 1 

b. 3 /tT 

c. 2 /tT 

d. 0 

e. none of the above. 



17. For the Rosenthal and Ferguson post hoc procedures 



a. the observed variances and covariances of the 

ranked data are employed to determine the 
confidence intervals. 

b. the - theoretical variance based on the uniform 

distribution is used to determine the confix 
dence intervals. 

c. one must have independent groups so that 

covariances need not be computed. 

d. the covariances of the original variables 

must be equal. 

e. the covariances of the ranked variables must 

be equal. 

l6. In performing an experiment when comparing two samples, 
let one sample contain 2 subjects and the other 5. 
subjects. Assume ranks from 1 to 7 are to be assigned 
to scores of the two samples. How many combinations 



o 



128 










19. 



of ranks 



3 • 

b. 

c. 

d. 
e • 



can the sample with 5 subjects take on? 



7 

10 

35 

1,2 

none of the above* 



Let E(T) 'sx ~ f (n-i 4 n ? 4-1) where T is the Wilcoxon 
statistic* If the Mann-Whitney statistic U equals. 

T - —L* (n^4- l) where n^ — 4 and n « 9, find the E(U). 

a. 10 

b. 20 

c. 36 

d. 45 

e* none of the above* 



20* For a given N, the expected normal order statistics 
form a 

a* normal distribution, 
b* Chi-square distribution* 

c. t distribution* 
d* rectangular distribution, 

e. none of the above* 

21. If in the Bell-Doksum test the sum of the * random normal 
deviates for one sample is denoted by L, then the sum 
of the random normal deviates for the second sample is 




L 

-L 

indeterminate from the 
0 

none of the above. 



information given. 



22. With regard to the Van der Waerden test, the inverse 
normal scores are more commonly referred to as 

a. Z-scores* 

b. t-scores. 

c. random deviates. 

d. W-scores. 

e. none of the above. 
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23. The small sample post hoc procedure of Tuback, Smith, 
Rose, and Richter for the Kruskal-Wallis test is 

a. valid for the set of all possible contrasts. 

b. controls the error rate by contrasts. 

c. valid only for simple contrasts, 
d© requires a re-ranking of the data. 

e. is less powerful than the procedure that 

employs C - l-»O as a multiplying 

constant© 



24. The Kruskal-Wallis test 

a. requires special tables for small samples, 
b© has a power efficiency . of 95% when compared 
to the one way analysis of the F-test. 
c. requires independent observations between 
and within samples, 
d© all of the above© 
e© none of the above© 



25© The large sample approximation for the Terry-Hoeffding 
and Van der Waerden Normal Scores tests require that 
the sample size of both samples be 

a. greater than or equal to 

b. greater than or equal to 10. 

c. greater than or equal to 17 . 

d. greater than or equal to 30. 

e. none of the above. 




APPENDIX B 



The reprints of journal articles handed to participants were 
the followings 

Cohen, Jacob, "An Alternative to Marascuilo ? s ’Large-Sample 
Multiple Comparisons 9 for Proportions,” Psychol, Bull, 67 
(1967), 199-201. 

Kruskal, W. H. and Wallis, W. A. "Use of Ranks in One Cri- 
terion Analysis of Variance.” Journal of the American Statis 
tical Association 47, (1952), 5&3-621. 

Marascuilo, Leonard A. ** Large-Sample Multiple Comparisons,” 
Ps ychol, Bull, 65 (1966), 280-290. 

Marascuilo, Leonard A. and McSweeney, Mary e lien. "Nonpara- 
metric Post Hoc Comparisons for Trend,” Psychol . Bull, 67 
(1967), 401-412. 

Rosenthal, Irene and Ferguson, Thomas. "An Asymptotically 
Distribution-Free Multiple Comparison Method with Applica- 
tion to the Problem of n Rankings of m Objects,” British 
Journal of Mathematical and Statistical Psychology 18 
(1965), 243-254. 

Wilcoxon, Frank. "Individual Comparisons by Ranking Methods 
Biometrics Bulletin 1 (1945), 80-83. 
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DESIGN AND ANALYSIS OF 
COMPARATIVE EXPERIMENTS IN EDUCATION 




Director 

Dr. Jason Millman 
Cornell University 
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INTRODUCTION AMD 



OBJECTIVES 



- • + Th ® Presession on the Design and Analysis of Comparative 

Experiments m Education marks the third straight year that the U.S. Office 
of Education has supported a five-day training session prior to the annual 
meeting of the American Educational Research 1 Association. The experience 
an ideas which have come out of the previous presessions have contributed 
a great deal to the successes of the 1968 Presession. 

The general objective of the presession is to increase the 
competence of educational researchers in matters of the design and analysis 
of comparative experiments. This objective is reflected behaviorally by 
the tests, problem sets, and inventories which form a part of the instruc- 
tn.ona.1 materials. We strongly believe that increased effort needs to be 
made to get "good" research data and to use proper methods of analysis so 
that more rational decisions affecting educational practice can be made. 

In tins presession, we hope to impart this attitude and to provide skills 
and knowledge which will contribute to the realization of this goal in 
situations in which the conduct of comparative experiments is appropriate. 
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STAFF 



Director: 

Jason Millman - Cornell 
Department of Education 
Ithaca, New York 14850 

Full Time Instructor: 

Kenneth D. Hopkins - 
University of Colorado 
Laboratory of Educational 
Research 

Boulder, Colorado 80302 

Part Time Instructor: 

Donald T. Campbell - 
Northwestern 
Department of Psychology 
Evanston, Illinois 60201 

Part Time Instructor: 

Nathan L. Gage - Stanford 
School of Education 
Stanford, California 94305 

Graduate Assistant: 

Bruce M. Gansneder - Ohio St. 
School of Education 
Columbus, Ohio 43210 

Graduate Assistant: 

Percy Peckham - 
University of Colorado 
Laboratory of Educational 
Research 

Boulder, Colorado 80302 
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PARTICIPANTS 



A. Selection of the Participants . 

Applications were received through the office of the overall 
presessions director, Gene V. Glass. The principal criteria for selection 
were that applicants had taken at least one course in statistics, had 
indicated reasons for attending the presession which were consistent with 
the objectives of the program, and had received a doctorate. A few 
doctoral students were admitted, however, for special reasons. 

Individuals who seemed overqualified were each written a personal 
letter raising the question of whether they might find it more profitable 
to attend a different presession. All applicants who were rejected were 
written a personal letter explaning the basis for the rejection. 

A total of 99 individuals applied, of which 64 were accepted 
and attended the meeting, 24 were accepted but later dropped out because 
of conflicting obligations, and 11 were rejected. 



B> Correspondence with Accepted Applicants . 

The following letters were sent to all accepted participants. 
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1968 AERA Presession on the Design 
and Analysis of Comparative Experiments 

To: Applicants Selected to Participate, in the Design and Anaylsis Presession 

From: Jason Millman and Kenneth D. Hopkins 

Date: December 8, 1967 



It is our pleasure to inform you that you have been chosen to be 
invited to participate in the 1968 AERA Design and Analysis Presession. The 
dates of the presession are from early Saturday morning, February 3 through 
Wednesday noon, February 7. 

If for any reason you find, it impossible to attend the Presession 
for the full five davs . please telephone or write immediately to withdraw so 
that one of the several excellent alternates we had to turn down can be given 
your position. 

The Presession will be held at the Abbey Resort Hotel on Lake Geneva, 
Fontana, Wisconsin. Our successful experience last year with a similar type 
of hotel in the New York city area influenced greatly our decision to choose 
the Abbey. We feel fortunate to be able to arrange for a group rate of $,£.2.54 
per day per person. This charge includes occupancy in a double room, meals, 
gratuities, sales tax, and access to the recreational facilities — -many of 
which are free or have but a nominal charge. 

Participants should check into the Abbey during the afternoon or 
evening of Friday, February 2. Chartered bus transportation from the OSare 
airport (Chicago) is being arranged in cooperation with the directors of 
another presession also being held at the Abbey. Buses will leave 0'Har>e 
over an extended period of time from early Friday afternoon to late that 
evening. Return transportation to Chicago is also being arranged. Cost will 
be in the neighborhood, of a few dollars each way. Details will be discussed 
in another mailing which we shall send to you during the latter half of 
January. 



It will not be necessary for you to make your own reservations at 
the Abbey; you need only check in on February 2, 1968. All participants 
will be placed in double rooms so that the expense can be held to the amounts 
quoted above. In the January mailing a list of participants will be included. 
You may at that time, indicate to us if you have a preference for a roommate. 

We shall rely almost entirely on specially prepared instructional 
materials. The materials which you will be given at the Presession should 
be reasonably self-sufficient, though you may wish to bring along a few refer- 
ence works (including an elementary statistics book) from your personal library. 
Come prepared to work. 

We would hope that you could bring to the presession Experimental and 
Q.uasl-Experimental Designs for Research by Donald T. Campbell and Julian C. 
Stanley. Rand-McNally, 1966. I believe it sells for $2.00. The bulkier Hand - 
book of Research on Teaching^ (N. L.Gage, Editor) is equally satisfactory since 
the paperback is merely a reprint of Chapter 5 of the Handbook . 



We recommend that prior to the February presession you read the first 
2 Pages in the Campbell-Stanley chapter mentioned above. In addition you 

Should be familiar with the basic ideas behind classical tas^of statistical 
hypotheses as described in the accompanying. paper. statistical 

. We are honored that you have indicated a willingness to take five days 

1L T WS are SUre iS 3 Very bus ^ schedule and spend it wita us! In 
turn, we are putting a great deal of effort into the preparation S this 

?o e !ou S1 °L^ lt wiU 1)6 a highly profitable experience 

to you. Wo look forward to working with you. 

attachment 
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1968 AERA Presession on the Design 
and Analysis of Comparative Experiments 



To: Applicants Selected to Participate in the Design and Analysis 

Presession 

From: Jason Hillman and Kenneth D. Hopkins 
Date: January 17# 1968 



This is our second, and should be our final, general mailing 
prior to the actual presession. Let us call to your attention two 
enclosures • 

First, there is a description of transportation accommodations 
from Chicago to the presession site. This schedule has been worked 
out in cooperation with the other two presessions which are now also 
scheduled to be held at the Abbey. Dinner will be available at the 
Abbey on Friday evening (2/2) if you arrive while the main dining 
room is still open. 

The second enclosure is a list of participants thus far 
expected to attend. Please indicate to us by January 26 at the 
latest if you have a preference for roommate. Should you wish to 
occupy a single room and are willing to pay the extra cost, please 
let us know this by the 26th of January also. Lacking any statement 
of preference, you will be assigned a roommate with somewhat common 
interests or position as indicated in the original application. It 
is not necessary for you to make your own reservation at the Abbey; 
you need only come and check in. 

We are hard at work preparing individual notebooks which will 
contain close to 200 pages of specially prepared instructional 
materials, including expository papers, working papers, problem 
sets, and even paper on which to take notes. All you need to bring 
are some pencils, lots of energy, and, if convenient, the references 
listed in our first memo. 

As is true with most of the presessions, we shall have a 
morning, an afternoon, and an evening session. Nevertheless, there 
will be approximately an hour and a half prior to dinner during 
which you may wish to go swimming, to go ice skating, or to enjoy 
the other recreational diversions available at the Abbey. Casual 
sports wear will be quite appropriate for the daytime sessions. 

Abbey regulations require sport or suit coats for men and street 
dresses for women at the evening meal. 

We admit to a growing excitement as the presession draws close. 
Try to leave your troubles and work at home; we* 11 have sufficient 
problems to occupy you. 

Attachments (2) 

i 

co: Donald Campbell 
Nathan L. Gage 
Bruce H. Gansneder 
Perc Peckham 

211 Stone Hall, Cornell University 
Ithaca, New York 14850 
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I. If you arrive at CM Hare airport . 

Pick up your luggage and go to the circular restaurant building, 
centrally located at the airport, where a private room has been 
reserved for your convenience while waiting for chartered bus 
service to the Abbey, Liquid refreshments will be available. 
Specifically, the full name of the room reserved is Seven 
Continents Restaurant's VIP Meeting Room, Thi3 meeting room 
(not' to be confused with VIP rooms of specific airline 
companies) is on the second floor of the restaurant building. 

It is suggested you use the one escalator in the building 
located in the center of the rotunda. 

Subject to possible change, the reserved (yellow school? buses 
are tentatively scheduled to leave from just outside the 
restaurant building from the lower of the two street levels at 
2:30, 4:00, 7:00, 8:00, 9:00, and 11:30. One way cost will be 
approximately $3«00. 

II. If you drive. 

m mmm m m * i tm — 

The Abbey is located in Fontana, Wisconsin just across the 
Illinois border and on the southwestern edge of Lake Geneva. 

The locality of Walworth is one mile from the Abbey. 

Ili, If you are in downtown Chicago . 

Take a Milwaukee Road commuter train at Union Station to 
Walworth, Trains leave 9:30 a,m,, 4:40 p Ira, 1 , ^anc“7> : 30 p #m . and 
arrive in Walworth 75 to 90 minutes later. (The 4:40 train 
does not operate on Saturdays or Sundays • ) One way fare is 
listed at ^2.77. If you notify them, Abbey personnel will pielc 
you up at the Walworth Station. 

Alternatively, you can take a Continental Air Transport Bus from 
the Monroe Street entrance of the Palmer House (Hotel) to the 
circular Restaurant * * building and go to the Seven 

Continents Room as indicated in I. above. 

IV. If weather forces O'Hare down and roads to the airport are 
blocked. 



We shall endeavor to divert the chartered buses to the Monroe 
Street entrance of the Palmer House, Call the Chicago office 
of the Abbey, Area Code 312-332-3586, for instructions. You may 
not get any satisfaction, but you should get lots of sympathy. 

Note: Chartered buses to downtown Chicago leaving the Abbey 
Wednesday after the noon meal are being arranged. 
Information will be provided during the presession 
meetings. 



* We are indebted to Andrea R. Rozran for supplying us with the 
information contained on this page and for making the necessary 
transportation arrangements. 
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C. Description of the Participants 



The 64 participants came from 26 different states, three provinces 
in Canada, and Stockholm, Sweden. By far the most participants came from 
New York State. 

Fifty-seven of the 64 participants were men. The median age was 39, 
with 34 and 43 being the first and third quartiles. The median year in which 
the doctoral degree was awarded was 1964* with 1959 and 1967 representing the 
first and third quartiles. Fifty-five of the participants indicated that this 
was the first AERA presession they had attended. 

The median scoring participant devotes 50% of his time to research 
and has authored five research articles or technical reports. 



D. Names and Addresses of Participants 



Raymond J. Agan 
103 Holton Hall 
Manhattan, Kansas 66502 
(Kansas State University) 

James G. Anderson 
Research Center, Box 3Y 
University Park, N.M. 88001 
(New Mexico State University) 

Ronald D. Anderson 
School of Education 
Boulder, Colorado 80302 
(University of Colorado) 

Maurice C. Barnett 

1300 University Club Bldg. 

Salt Lake City, Utah 84III 
(Utah State Board of Education) 

Charles P. Bartl 
College of Education 
Reno, Nevada 89507 
(University of Nevada) 



Leon W. Bonner 
P.0. Box 303 
Normal, Alabama 35762 
(Alabama A & M College) 

Bar tell W. Cardon 
Ed. & Psych. Center, Bldg. 2 
University Park, Penna. 16802 
(Penna. State University) 

June R. Chapin 
1190 Bellair Way 
Menlo Park, California 94025 
(College of Notre Dame, on leave 
from University of Santa Clara) 

James W. Colmey 
School of Education 
Memphis, Tennessee 36811 
(Memphis State University) 

Richard J. Colwell 
406 W. Michigan 
Urbana, Illinois 61801 
(University of Illinois) 



Julianne L. Conry 
1311 Morrison Street 
MadiBon, Wisconsin 53703 
(University of Wisconsin) 



Shen Henrysson 
School of Education 
Stockholm 34 , Sweden 
(University of Stockholm) 



Harold Cook 
414 Huntington Hall 
Syracuse , New York 13202 
(Syracuse University) 



Michael J. A, Howe 
Department of Education 
Medford, Mass. 02155 
(Tufts University) 



Arvin D. Crafton 
801 Sycamore Street 
Murray , Kentucky 42071 
(Murray State University) 

Anant S . Deshpande 
310 Administration Bldg. 
Atlanta, Georgia 30332 
(Georgia Tech.) 

Calvin O. Dyer 

3002 uhs 

Ann Arbor , Michigan 48104 
(University of Michigan) 

Herbert Garber 
School of Education, U*»4 
Storrs, Connecticut 06268 
(Univ. of Connecticut) 

Allan W. Gibson 
6833 Kingston Place 
Tucson, Arizona 85710 
(Amphitheatre Public Schools) 

Frank P. Greene 
707 Sumaer Avenue 
Syracuse, New York 13210 
(Syracuse University) 

Edward V. Hackett 
Bureau of Fduc. Research 
Kent, Ohio 44240 
(Kent State University) 

Judson A. Harmon 
3263 N. Shepard Avenue 
Milwaukee, Wise. 53211 
(Univ. of Wisc-i Milwaukee) 



John L. Hughes 
Dept. 914, South Road 
Poughkeepsie, N.Y. 12602 
(IBM) 

William H. Johnson 
Dept, of Bduc. Admin., Box 3R 
University Park, N.M. 88001 
(Mew Mexico State univ.) 

Jean-Marie Joly 
Department of Edonation 
917 Mgr. Grandin 
Quebec 10, P.Q. 

(Institute of Research in Educ.) 

Evan R. Kei&lar 
School of Education, 303 Moore 
Hall, 405 Hilgard Avenue 
Los Angeles, California 90024 
(UCLA) 

John J. Kennedy 
College of Education 
Knoxville, Tennessee 3f916 
(University of Tennessee) 

Jeremy Kilpatrick 
Dept, of Math. Education 
New York, New York 10027 
(Teachers College, Columbia) 

Martin Kling 

Graduate School of Education 
10 Seminary Place 
Mew Brunswick, N.J. 08903 
(Rutgers-The State University) 



Frederick G. Knirk 
121 College Piece 
Syracuse, Hew York 13210 
(Syracuse University 

John R. Kolb 
3303 Leonard Street 
Raleigh, north Carolina 27607 
(Ho* Carolina State Univ • ) 

Wilbur K, Kraybill 
305 E. Pairview Avenue 
Altoona, Pennsylvania 16601 
(Penn, state Univ* at Altoona) 

Leonard H. Kreifc 
Bduc. Research Program 
14th Avenue & Lake Street 
San Francisco, Calif, 94118 
(Dental Health Center, Public 
Health Service) 

Lisa R, Kuhmerker 
81*14 Austin Street 
Raw Gardens, Hew York 11415 
(Hunter College) 

Robert E. Leibert 
Reading Center, Room 286 
4825 Troost 

Kansas City, Missouri 64110 
(univ, of Missouri at Kansas 
City) 

H, Keith MacKey 
Box 578 

Halifax, Nova Scotia 
(Nova Scotia Dept, of Bduc,) 

Lester Mann 

443 S, Gufkph Road 

King of Prussia, Pa, 19406 

(Montgomery Co, Public Schools) 

Rosemarie B. McCartin 
404 Miller Building 
Seattle, Washington 98105 
(Univ, of Washington * on 
leave from Seattle univ.) 



Thomas E. McCloud 
19955 Lauder 
Detroit, Michigan 48235 
(way ne Co, Child Development 
Center) 

Arthur s, McDonald 
2 Richards Drive 
Dartmouth, Nova Scotia, Canada 
• Cf-S] pt. of Education, Province 
of Nova Scotia) 

Richard J, McLeod 

B-37 McDonel Hall 

Science & Math, Teaching Center 

Bast Lansing, Michigan 48823 

(Michigan state Univ,) 

James M, Moser 
3415 N. 95th Street 
Boulder, Colorado 80302 
(university of Colorado) 

Jerome Moss, Jr, 

Dept, of Industrial Education 
Minneapolis, Minnesota 55455 
(University of Minnesota) 

Ovle H, Old ridge 
Faculty of Education 
Vancouver, British Columbia 
(Univ, of British Columbia) 

Angela Pace 
4 Cowance Street* 

Cortland, New York 13045 
(State Univ, of N,Y, at 
Cortland) 

Paul T. Rankin, Jr, 

30130 Wicklow Road 
Farmington, Mich. 48024 
(Detroit Public schools) 

Joseph Reswick 
520 B. 21 Street 
Brooklyn, New York 11226 
(N.Y,C. Board of Education) 
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Barak V. Rosenshine 
Ritter Hall 

Philadelphia, Pa. 19122 
(Temple University) 

Donald R. Senter 
Huntington, Hew York 11743 
(Educational Developmental 
Laboratories , Xnc • ) 

Bernard J. Shapiro 
School of Education 
765 Commonwealth Avenue 
Boston, Mass. 02215 
(Boston University) 

John L. Shultz 
School of Education 
St. Louis, Missouri 63121 
(Univ. of Missouri at 
St. Louis) 

Theodore C. Sjoding 
705 S. 121st Street 
Tacoma, Washington 98444 
(Pacific Lutheran University) 

Gerald P. Speckhard 
Valparaiso, Indiana 46383 
(Valparaiso university) 

Irving S. Spigle 
242 South Orchard Drive 
Park Forest, Illinois 60466 
(Park Forest Public Schools) 

Robert M. Smith 
Special Education Bldg. 
University Park, P^. 16802 
(Penn. State university) 

William R. Stewart 
255 Chambers Bldg. 

University Park, Pa. 16802 
(Penn. State university) 

Howard W. Stoker 
413 Education Bldg. 
Tallahassee, Fla. 32306 
(Florida State univ.) 



Jerome Taft 
1410 HE Second Avenue 
Miami, Florida 33132 
(Dade County Schools) 

Harriet Ta Image 
1011 N. Hayes Avenue 
Oak Park, Illinois 60302 
(Univ. of Illinois at Chicago 
Circle) 

- i 

Donald R. Thomsen 
3808 Horriswood 
Memphis, Tennessee 38111 
(Central Midwestern Regional Ed. 
Lab. & Memphis State Uhlv.) 

Virginia H. Vint 
7 Brookwood Drive 
Hozmal, Illinois 61761 
(Illinois State University) 

John L. Wasik 
3712 Horton Street #104 
Raleigh, north Carolina 27607 
(Worth Carolina State Univ.) 

Theodore M. JSink 
Glassboro, Hew Jersey 08028 
(Glassboro State College) 
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Rrmqh Quid© to Research . J&t er.ei»t .s,. &£ — Po xir_l, c_^J2l^I!L§. 



Aflrrinly t rat ion and/or 
Qgqanizat lone 
Colmey 
Johnson 
Neff 



Bisadv ant acred .andi-O r 
slov Learne r 
Harmon 
McCartln 
Rankin 
Smith 



Attitudes 

Chapin 

Harmon 

Hines 

K&ltfi*wnis 

Vint 



feiarlv Child, &!«, 
Kuhroerker 

Education in the A^ts 
Colwell 
St ewart 



C omputers 

Chapin 

Taft 

C reativity 

Harmon 

Culture and E ducation 
J. Anderson 

Curr ionium/ Inst ruct ion 
~ Leibert 

Mann 

McDonald 
Moss 

Old ridge 
Senter 
Spigle 
Smith 
T&lmage 
E ink 



Agan 

Barnett 

Chapin 

Colwell 

He^shpande 

Garber 

Greene 

Johnson 

Keislar 

Knirk 



Vint 

Hi gher Educati on 
Chap If* 
Kaltsounis 



lear ning : (W ct **X l*<?«rning f 
Concept Development j Child 
Development and. Teaming? 
Memory) 

Con rey KuNae rko r 

CC 5k 

Greene MeCartin 



He- islar old r i ?ga 

Kennedy aoseashine 
Kling Shapiro 

Kolb **asik 



M ath education 
Dyer 

Kilpatrick 

Kolb 



r>n«iov t)t Measurement and/o r 
Evaluation 



Moser 

Pace 



J . Anderson 


Hughes 


Sjoding 




Barnett 


Joly 


sp^ckhard 


Media 


Bart! 


Kennedy 


Spigle 


Kling 


Bonne r 


Kreit 


Stoker 


Knirk 


Crafton 


MacKay 


Taft 


i 

Motivation 


fteshp ande 


Moser 


Tallage 


Gibscn 


Re swiok 


Thomsen 


Alter 


Hackett 


Senter 


Wasik j 


McClo 


F^nryrwon 

Hines 


Shapiro 


Zink ! 

1 

I 





Personality 

Alter 

Dyer 

Hackett 

McCloud 

Shultz 

Physical Education 
Mann 

Physiological Indices 
Bartl 
McCloud 

Problem Solving 
Harmon 
Kilpatrick 
Wasik 

Pupil Personnel Progress 
Kraybill 
McDonald 
Shultz 

Dyer 
Kling 
Leibert 
Rosen shine 

Science Education 
R. Anderson 

Social Peviancv 
Zink 

Teacher Behavior 
Rs Anderson 
Chapin 
Crafton 
Harmon 
McLeod 
Rosenshine 
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SCHEDULE 


l T\ Day Session 

JJ 


Topics and Principal Instructors^ 


t * 

A.M. 

S 


Elementary design (M); nature of control (M); types and functions 
of variables (M); internal validity with emphasis on regression 
(M,H). 


A “ /ft. 


Internal validity (con't); interactions (M); external validity (ft). 


T 

Eve. 


External validity (con’t); relation of statistics to experimental 
design (H,M); functions of analysis (H,M). 


s A - M - 


Introduction to classical analysis of variance model (H). 


U Aft. 


Rules of thumb for the analysis of complex, but balanced, experi- 
mental designs (M). 


N Eve. 


Rules (con't); question period. 


m A.M. 

M 


Quasi-experimental designs (C). 


0 Aft. 


Quasi -experimental designs (.con't); analysis of unbalanced designs 
or designs in which AMOVA assumptions are not met (H). 


N Eve. 


Violations of assumptions, con't (H,P); question period; party. 


ip A .M . 

u 

g Aft. 

s 

Eve. 


Analysis of covariance (H)« 

Multiple and planned comparisons (M,H) 

Causal relations from correlational data (G) ; question period. 


W 

E A.M. 

D 


Interface between theoretical orientation and research design (G); 
Participant generated problem(s) in which an appropriate design 
and/or analysis plan needs to be formulated (G, M, H); participant 
attitude inventory. 




— " , ' ' i 

£ 



* M s Millman; H => Hopkins; C * Campbell; G 88 Gage; P * Peckham. 



U 
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STUDY AND REFERENCE MATERIALS 



Each staff member brought with him a selection of material supple- 
mentary to his lectures. Each participant, was asked to bring Experimental 
and Quas i -Experimental Designs for Research by Donald T. Campbell and 
Julian C. Stanley. (Rand-McNally, 1966). Prior to coming to the Presession, 
each participant was mailed a copy of the first item listed below under - 
"Materials’’ 

Upon arrival at the presession site, each participant received a 
loose-leaf notebook which contained all the materials (demonstration and 
expository papers, problem sets, evaluation instruments, scrap paper) he 
needed. With the exception of items 17, 19, 22, 23, 25, 32, and 38, these 
materials are unpublished. A list of these materials follows. 



Materials Used in the 1968 Presession on 
Design of Experiments 



1. Introductory general remarks about inferential statistics, classical 
procedures. Jason Millman. Pi 5. 

2. Bibliography: design and analysis of experiments in education. Jason 

Millman. P: . 7. 

3. Randomization demonstration. Jason Millman. P. 1 . 

4. Poem demonstration. Jason Millman. Pi. 4 . 

5. The design of experiments - some elementary concepts. Jason Millman. 

Pi 15. 

6 . Experimental treatments. Jason Millman. P, 2. 

7. Notes on sources of internal experimental invalidity. Gene V. Glass. 

K 2 . 

8. Instructional paper on regression and the matching fallacy. Kenneth D. 
Hopkins. P, 7. 
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9. 11 ttt tit rat ions of sources of internal experimental inval- 

idity* Gene V Glass* Pi ; 2. 

10* Quia on elementary concepts of design and sources of 
internal invalidity* 8 test items. *>; 3 - 

11. ANOVA interactions in factorial designs. Jason Hillman. 

Py> . 8. 

12. Problem set to accompany ANOVA interactions in factor- 
ial designs. Jason Mi 11m an. P; 5. 

13. The external validity of comparative experiments in 

education and the social sciences. Glenn H. Bracht 
and Gene V Glass. P* 13. , 

14. Two untitled pictures. Ho author. P. 1* 

15. CLUG Experiment. Jason Hillman. Py ► 2. 

16. Quiz on interaction and sources of external invalidity. 

7 test items. P> 3. 

17. Rules of thumb for writing the ANOVA table. Jason 

Hillman and Gene V Glass. P$, 11. 

18. Problem set to accompany "Rules of thumb..." Jason 

Mi liman. 11. 

19. BMD08V Analysis of variance. Ho author. P* 6. 

20. Quiz on rules of thumb for writing the ANOVA table. 

4 test items. P, 2. 

21. Quasi-experiments in education. A 75 minute paper read 
to the participants by Donald Campbell. A final version 
(to be distributed to all participants) is now in prepar- 
ation. 

22. Quasi- experimental designs. Donald T. Campbell. P* 4. 

23. Administrative experimentation, institutional records, 
and nonreactive measures. Donald T. Campbell. P* • 12. 

24. Conseguences of failure to meefl the assumptions under- 
lying the analysis of variance. Gene V Glass., P* 12. 




25* 



V C| 3 * 



28* 

27. 

28. 
29. 

SO. 

31. 



f ing k^Miog-aneity of variances. 



2. 



Computational illustrations - b 0 m 09 e.ne.tty ox variance 

t«s t*. Kenneth Hopkins* Pp . 2. 



The analysis of repeated measures designs* Gene V Glasft 
and Kenneth Hopkins. P, 6. 

Program epsilon. Percy D. PecJcham. P. 1. 



The axperimental unit in statistical analysis. Percy !>. 
Peckham and Kenneth B. Hopkins. P, 5. 

Violating ANOVA assumptions — problem set. Kenneth P. 
Hopkins. P^ 2. 

Wuiz on violating A NOV A assumptions. 8 test « Pv 



32. Analysis of covariances its nature and uses. William G. 
Cochran. P» 7. 

33. AUCCVA lecture problem. Kenneth P. Hopkins. Pj. 5. 

34. Analysis of covariance - problem sets. Kenneth D. 
Hopkins. P. 1. 

35. ANCOVA computational problem sets. Kenneth 0. Hopkins. 

P\, 7. 

36. Quiz on AHCOVA. 8 test questions. P 2. 

37. Xxidividual comparisons among means. Jason Millman. Pit 

38. A schema for proper utilization of multiple comparisons 
in research and a case study. Kenneth D* Hopkins and 
Russell A. Chadboum. P* 6. 



39. Worked examples of multiple comparison procedures u?*ing 

data irom the Rothkopf study. Jason Hillman. P 2 * 4. 

40. Problem set on multiple comparisons. Jason Hillman. 

P «y 4 * 

41. Multiple comparisons for correlation coefficients?, 
proportions, and other parameters. Kenneth 0. Hopkins • 

Pi. 3. 

42. Quiz on multiple-comparison procedures. 9 test items. 

^ 3. 
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43 p*tcn»iidj*j causal relationships If- « w.-i'-v- r a v choice i c » 1 

inquiry* four techniques. A. a. ye® ana N. i>. Caq®. 

P* / 5 • 



44. participant evaluation form, P# 2 » 

45. Preaeaaion critique for staff neniberae 2* 
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OUTCOMES 



A. Subject j 

At regular intervals during the presession short quizzes were 
administered. The participants recorded their answers on a separate cover 
sheet which they handed in. They were then able to keep the test proper 
while the questions and answers were discussed by the instructional staff. 

Unlike previous years, these tests were not mastery examinations. 
Rather, they were composed, of more difficult items which served as a 
learning experience. Indeed, in some instances the content necessary to 
know how to answer a question correctly had! not yet been provided. 

The proportion of participants who answered a given item correctly is 
shown below. 



Test 



Item 






A. 



U 



Intro. - 
Internal Valid. 


58/61 


44/61 


31/61 


Interaction - 
External Valid. 


41/64 


42/64 


53/64 


Rules of Thumb 


Not Scored 




Violating ANOVA 
As sump. 


50/62 


7/62 


53/62 


ANOVA Quiz 


19/63 


53/63 


26/ 63 



18/61 

j 


36/61 


49/61 


19/61 


34/61 


li/64 


25/64 


13/64 


— 


— 


41/62 


32/62 


35/62 






46/63 


63/63 


42/63 


60/63 


37/63 



45/61 - 42/61 



Multiple - Com- 
parison Quiz 



37/61 



47/61 



52/61 



17/61 45/61 18/61 
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Ihe instructional staff were pleased with the test results in most 
.instances. The participants were not expected to absorb all of the instruc- 
tion. Rather, j" t was hoped that the wealth of instructional materials plus 
the instruction would provide the basis for further study and reference. 



As will be seen in the next section, many participants were disappointed 
that they coulcj not master all the content. Whether future presessions should 
try to cover less material at a slower pace or not is a real issue which 
deserves further consideration. (A recommendation on this matter may be 
found in Section VII-B-5.) j. 



B. Attitudes 

A participant and a staff critique form were filled out. The replies 
k> each question on the participant critique sheet are provided below, together 
with last year's responses whenever the same question was ashed. 



PARTICIPANT CRITIQUE 

1. a. -To what extent did the relative availability or unavailability of boohs 

and , Journals interfere with or promote your attempts to master the 
content of this session? 

Thirty-seven respondents said no need for boohs and journals 
since content was provided by notebooks of materials. Seven said 
availability wasn't a question - time to read them was. Ten pointed 
out that boohs were not available and might have been helpful. 

b. To what extent did reproduced materials given to you by the staff 
improve matters? 

Forty-two respondents replied "Very much." Eleven more also 
said "very much" but lamented the fact that there wasn't more time 
to read them. Four replied that some materials were better than 
others . 

2. a. Did you feel that you lacked a "place to work," either alone or in 

small groups? 

Yes - Somewhat - 8; No - 44 
Responses to this question last year were: 

Yes - 11; No - 43. 
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your room satisfactory? 

Yes - 43* Food bad or expensive — 16 
Responses to this question last year wares Yes - 43; Sto - 11 

Which features of the meeting rooms were inadequate or 

net conducive to learning? 

None - 6 ; lighting - 46; Acoustic* - 4; Cold - Si- 
Table arrangements - 6f Miscellaneous -2. 

Last year, acoustics was the problem most cited * 



Which features were especially facilitat&ve .in the same 



regard? 

Arrangement 
Instructors 
Last year, space 



of tables - 30; lighting - 2; Acoustics • 
- 3; Atmosphere - 2; Nothing outstanding 
and water were mentioned most often. 



3 



Was five days too long a period to lea^e your wjdfc at home 
for the purpose of attending this session? 

No - 47? Yes - 7; Exhausting - 3. 

Last year, 39 said No; 15 said Yes* 



Was five days too short a period in which to lear?is much 
of the content of this session? 

No *• 30; Yes *•* 23# 

Last year, 24 said NO; 27 said Yes. 

Were you allowed enough time in which to pursue activities 
of your own choosing? 

No - 19; Yes - 3ft. 

Last year, 25 said No; 28 said Yes. 

Would you have preferred not to meet in the evening after 
dinner? 

No - 44; Yes - 8; Sometimes - 9. 

Last, year, 37 said No; 16 said Yes. 

Would you have preferred more or fewer meetings per day 
than there actually were? Or was the number of meetings 
per day agreeable to you? 

O.K. as is - 47; Fewer - 9. 

Last year, 48 said O.K.; 6 said Fewer. 



6a. Were the individual lectures too long to sit and listen 
or twice notes? 

Mb •* 4?? Yes - 5? Sometimes - 7, 
tost year# 36 said No? 9 said Yes? 9 said Sometime*** 

6b, Were the lectures scheduled in an appropriate sequence? 

No - 4? Yes - 48i 

Last year, 4 said So? 44 said Yes* 

7. Did you have sufficient opportunity to interact with 
colleagues? 

Nb - 7? Yea - 52, i 

Last year, 9 said No? 42 said Yes* 

8a. Were the .instructors too in accessible or inapproachable 
so that you did not get the individual attention that 
you desired? 

Mb - 55? Yes - 2? More assistants needed - 3 . 

Last year, 52 said Mb? none said Yes. 

8b. Was it helpful to have graduate student assistant a present 
No - 2? Yes - 55? Could have .been used better - 2* 

9. Did the attempts to evaluate your pjxr.y. •es* and tactions 
during the session (and at this moment) interfere with 
your vo rife here? 

m - 55? Yes - 3* 

Last year 46 said No, 3 said Yes. 

10. In general, was the breeession well organised? 

No - o? Yes - 55? Undecided - 2. 

Last year, none said Mo, 49 said Yes. 

11a. Did the content of the lectures and readings pre#uppc!«e 
far mere previous training than you had? 

No - 39? Yes - 13? In some areas - 5. 

lib. Should less training in these areas or more have been 
presupposed? 

(Most respondents answered "Yes" or "No" to this* 
question. The meaning meant for these replies is uncl^&r. 

12. To what extent was the content ot the lecture;? and v^a-3- 
inge relevant to what you had hoped to accomplish during 
the seas ion? 

Relevant - 50? Not relevant • 4? undecided ~ 2* 

Last year, 48 out of reppendents said "Relevant." 
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13 a, the 1 

Yes - 
L-&<st year* 



efcurers* stimulating and internet ing? 
46? Sortie more that others - 1C, 

49 of 51 respondents said Yes, 



13b. Were the lecturers competent to speak on the subject 
assigned them? 

Yefi **“ 40? Some wore than others — 7? Undecided — 1. 

Last year, 44 said fee? 7 omitted the question. 



13c. Were the lecturers well prepared? 

Yes - 42? Some more than, others - 14 . 
Last year 59 out of 51 said Yes. 



For questions 15 - 21, last year’s frequencies are shown in 
parentheses when they were available* 

15. :i£ you had it to do over again would you apply for the 
Presession which you have just completed? 

Yes s 58 (49) Mb: JL11L. 

16. If a presession such as this is held again would you 
recommend to others like you that they attend? 

Yes: 59 J& iJL. JLiSJL 

17. l>o you anticipate maintaining some sort of contact with 
a t least oiKr of the Presesiion staff? 

Yes: 38 (43) Bto* 15 (8). Possibly: 

le. Do you feel that AER& is making an important contribu- 
tion to education by sponsoring precessions such as 
this one*; 

Yes: 59 (51) Mo: 0_ 1 Ql_ 

19. i)a you feel that anyth i>penec during these 

five da y# to make it more likely that you will leave 
your present position of employment? 

Yes: 3 (12) Mb: 51 (391_ 

29, is it likely that you will collaborate in research with 
someone else attending this rree&sslon (oth^r than those 
;. jU already ware likely to c i l labor ate with) ; 

Yes* 16 (22) Mo: 42 (27) Doubtful* JLA21. 
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21. Do you. f*eel that the staff should feel that it has accomplished its 
objectives during this five-day presession? 

Yes - 56; No - 2 



In this listing of responses it was not feasible to indicate all of 
the qualifications. More often than not, a "yes" or "no" was qualified by 
such phrases as "great", "very much", "and how", etc. Individual comments 
included; "This was one of the finest training experiences which I have 
ever gone through;" "Stimulating, rewarding, and frustrating, all . at the 
same t im e ; " "If I sound effusive it's because I mean it;’* and "This was a 
model for Presessions." 

It is not surprising that the participant-reactions were highly 
similar to those for last year's design presession in view of the overlap 
of staff and some materials. Perhaps the major negative criticisms (invariably 
made by a minority of participants) concerned the fast pace of the presession, 
the lack of material beforehand, the expense of room and board, the poor 
lighting in the meeting room- and the dissatisfaction with one of the guest 
instructors. (This latter reaction was, in the main, responsible for the 
non-positive replies to questions 13a, 13b, and 13c. The director must take 
responsibility for not structuring the instructor's assignment in enough 
detail and for letting non-productive discussion get out of hand.) 

As for the staff critiques, unsatisfactory checks were placed only 
after le (resource material, library) and 3f (adequacy of financial support). 
Poor lighting in the meeting room was also mentioned. Other than these 
problems, the staff critiques displayed much self satisfaction (perhaps too 
much so) with those of the graduate assistants being most complimentary. 
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PRESESSION VI 



CONCEPTS. CONTEXT. AND STRATEGIES FOR 
EVALUATION OF EDUCATIONAL PROGRAMS 



Director 
Dr. Robert Pace 

University of California-Los Angeles 




STAFF 



Director: 



Ins true tors: 



Graduate Assistant: 



C. Robert Pace 

University of California, Los Angeles 
UCLA 

Department of Education 
405 Hilgard 

Los Angeles, California 90024 



Rodney Skager 
UCLA * 

School of Education 
405 Hilgard 

Los Angeles, California 90024 

Marvin Alkin 
UCLA 

School of Education 
405 Hilgard 

Los Angeles, California 90024 

Louise Tyler 
UCLA 

School of Education 
405 Hilgard 

Los Angeles, California 90024 

Daniel Stufflebeam 
The Evaluation Center 
209 Oxley Hall 
1712 Neil Avenue 
Columbus, Ohio 43210 

Fred Stodtbeck 
The University of Chicago 
Department of Sociology 
1126 East 59th Street 
Chicago, Illinois 60637 



J. Eugene Grigsby (Research Assistant) 

10128 Regent Street 

Los Angeles, Californ ia 90034 



PARTICIPANTS 



Ammentorp, William 1-1. 
Route 2 

Cannon Falls, Minn* 55009 



Anderson, Myron L. 

E 1 erne n tar y E due a t ion 
Wisconsin State Univ. 
Oshkosh 

Oshkosh, Wisconsin 54901 


(Regional Lab) 

. Training Elom. Teachers 
Culturally Deprived 


Aronson, Judith S . 
100 Arundel Place 
St. J .c u is , Mo . 6 3 1 05 


(F d Projects) 
Title 3 -III, Arts 


bach, Jacob 

Bureau of Elementary &. 
Secondary F.dnr. , Rm ?001 


Consultant 


US0E, FOB #6 (else - So. 111. 
400 Maryland A vc. SW 
Wash. I).C. 2020?. 


Univ.) 


Beisinan, Gladys L. 

963 Br own Rond. 

R oc. he s te c , Me w Y or k , 14622 
(Center for Urban Educ.) 


(Fo d . Pr o j e c t. s ) 
Title I- III 
Urban Education 


Bell, Lo we 11 Et 
2.01 E. 38th St. 


(Program Kvaluati on) 
Fader al Programs 



Sioux Palls Public Schools 
Sioux Falls, South Dakota 5710 



Berman, Laurence Ho 
Testing Services 
Western Michigan University 
Kalamazoo, Michigan 49001 


(Program Eva luatJ on) 
Schoo 1 Pr ogr 


Bernstein, Allen 

MOREL, 3750 Woodward Avenue 

Detroit, Michigan 48201 


(Regional Lab) 

Sc he ’1 Eval. 

Teachers- S Indents Attitude 


Bish-.-y, Gerald D. 
Bureau of Research & 
Exami na tion Services 
Univ. of Northern. Iowa 
Cedar Falls, Iowa 


(Institute onal Evaluation) 
College Eval. 



BoJh nbachor , Joan K. (Program Eva leation) 

Cincinnati. Public Schools System wide 

Division of Evaluation Services School Eval. 

60S East McMillan Street 
Cincinnati, Ohio 45206 

Brim, Charles W. 

Dir. bureau Univ. Research 
Nor f. Vie r n 111 . Univ . 

DeKalb, 111. 60115 



Burke, James M. (Fed. Projects) 

Conn. State Dept, of Ed. Title I, III 

165 Capitol Ave . P.0. Box 2219 
Hartford, Connecticut 06115 



Car 1 sf.roai, Gerard K. 

1929 Lake. hurst Drive 
Olympis, Washington 98501 



Carr i gan, Patricia M, 

1220 Wells Street* 

Ann Arbor, Mich. 48104 
(Ann Arbor Public Schools) 



Cawelti, Gordon L. 

5454 South Shore Drive 
Chicago, 111. 60615 
(North Central Assoc.) 



Coffin, Edwin C. 

Monterey County Office of Ed. 
P.0. Box 851 
Salinas, Calif. 93901 
Superintendent 



Co nr y , R o b e r t F . 
Instructional Research Lab. 
202 State St. 

Madison, Wisconsin 53706 
(Univ. of Wisconsin) 



Cooper, Homer C. 

Social Science Research Instil. 
Univ. of Georgia 
Athens, Ge or g in 3060 1 



(Program Evaluation) 
Federal Projects 



(Program Evaluation) 
School Programs 



(Institutional Evaluation) 



(Fed. Projects) 
Title l-III 



(Fed. Projects) 
Title I, TIT. 



(Fed. Projects) 
Title I, III 
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Cooper, Mil cl reel Pivetz 
T" ; * 234 N. Irving Street 

v ‘ ' Arlington, Virginia 22201 

(District of Columbia) 



Cortrighl. Richard W. 
4805 Crescent Street 
Washington, D.C. 20016 



Curtin, John T, 

23761 Hauer Dr. 

St. Clair Shores, Mich. 48080 
(Detroit Public Schools) 

Dickens, Charles II. 

Office of Assoc. Dir. (Ed.) 
Wall. Science Foundation Rm.622 
Washington, D,C. 20550 



Doepkc, Mrs. Loretta 0. 

2360 Came lot Drive 
Bropkf in 1 c! , Wis cons i n 53005 
(Milwaukee Public Schools) 



Dolan, G, Keith 
5500 State College Parkway 
San Bernardino, Calif, '92407 
(State College) 

Eagle , Norman 
Education Center, 

Mt. Vernon Pub. Schools 
165 N. Columbus Ave . 

Mount Vernon, New York 



E d .-?n r d s , P 1 a u 1 i rte 
17 7 Drexc* 1 Ave. 
Lansdowne , Penn. 19050 
(Abington, Pa.) ? 



E Ivr e 11, A 1 b e r t. R . 

Pankov Road 

Durham, New Hampshire 03824 
(Univ. of New Hampshire) 



9 




U 1 mm —M t— 



(Fed. Projects) 
Title I- III 



(Adult Education) 



(Fed. Projects) 
Title I, III 



(Educational Program) 



(Fed, Proj^ ets) 
Title I- III 



(Program Evaluation) 
Educ. Programs 



(Program Evaluation) 
Educational Programs 



(Fed. Projects) 
Title III 
Indiv. Instruction 



(Program Evaluation) 

O.E.O. 



FJocco, Edvard C. 

65 Jesup Pod. Board of lid. 
Westport, Conn. 

(Westport Public Schools) 


(Program Evaluation) 


Flottraann 
2812 Pine 

Mattoon, 111. 61938 


(Fed. Projects) 
Title I-III 


Fortune , Jini C . 

Bureau of Ed Research & Serve. 
Memphis State University 
Memphis, Tennessee. 38111 


(Pr'ogram Evaluation) 


Freyles, Fanny C. 

Div. of Eval-Edpt. of Education 
Hato Rey, Puerto Rico 


(Fed. Projects 
Title I-III 


Hefferlin, JB L. 

Ins tit . of Higher Education 
Box 101, Teacher College 


(Program Evaluation) 
Educ. Programs 



Colombia Univ. New York N.Y. 

10027 (Teachers College, Columbia) 



Hitch, Kenneth S. Colonel USA 
Indue trial College of the 
Armed Forces 
Fort McNair, Wash. DC 


(Program. Evaluation) 


House , Er rc s t R . 

Coo pc r a 1 5 ve E du ca t i ona 1 
Research lab. 

Box 815, Korthf ield. 111. 60093 


(Gifted) 


Hunter, Larry 0. 

Research & Inf or. Servs.Div. 
Unified School Dist. #259 
428 So. Broadway, 

VJichita, Kansas 67202 


(Fed. Projects) 
Title I-III 


Iwano t o , D av id 
US Office of Education 
400 Maryland Avenue S.W. 
Washington, D.C. 20202 


(Fed. Projects) 
Title I- II- III 

1 


Jenkinson, Marion E. 

Ontario Ins tit. For Studies 


• 



in Education 

102 Bl.oor St * V. Toronto, 5, Canada 
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Johnson, Thouas P. 

3820 Berkeley Drive, #3 
Grand Forks, Forth Dakota 58201 
(Univ. of llorth Dakota) 



Ken cl i g > Thom ns E. 

Div. School F.val. 

Rm 539 F.d . Bldg. 

P10. Box 911 
Harrisburg, Penn, 17126 
(Dept, of Public Inst.) 



Krahmer, Edward F. 
Bureau Ed. Research, 
Univ. of Horth Dakota 
Grand Forks, U.D. 58201 



La j . ng , J a i i'ii s H . 

75 Santa Barbara Road 
Pleasant llill ,CnlJ.f . 94523 



Lank ton, Robert S. 
Research. & Dev. 

Detioit Pub. School 
5057 Woodward 
Detroit, Michigan 48202 



Levy, Rafael A. 

710 E. Algonquin, Apt. Ill 
Arlington Heights, 111. 60005 



Mahan, David J. 

6733 West Park 

St. Louis, Missouri 63139 



Mazur, Joseph L. 

1055 Cl if dale 
Lakewood, Ohio AA 167 
(Cleveland public Schools) 



Me As ban, F i 1 d r e th It » 

2036 l, Vi 18th Lane 
Gainesville, Florida 32601 
(Reg. Lab) 

McCall on, Earl L. 

P.0. Box 138/1 

North Texas State Univ. 

Denton, Texas 76203 



(Fed. Projects) 
Title I- III 



(Program Evaluation) 
School Programs 



(Fed. Projects) 
Title I- III 



(Fed. projects) 
Title I- III 
Educ. Planning 



(Program Evaluation) 
School Evaluation 



(Indiv. Pres. Inst.) 



(Title III) 



(p r o gr am E v a 1 u a t i o n ) 
Fed. Projects 



(Fed. Projects) 
Title I-III 



(Fed. Projects) 
Title I- III 



McDonald, Art mix S. 

2 Richards Dr. Dartmouth 
Nova Scotia, Canada 


(Pupil Personnel) 


Keinko, Dean L. 

15 South Fawn Drive 
Terre Haute, Indiana 
(Indiana State Univ.) 


(Fed. Projects) 
Title I- III 


Meredith, Gerald M. 
East- We st Center 
1777 East- We st Road 


(Institutional Evaluation) 
Institutional Evaluation 


Honolulu, Kavxai i 96822. 
(University of Hawaii) 


# 


Miller, Harold J. 

713 Nor tb we s t Dr ive 
Grand Forks. Nprfh Dakota 
58201 

Univ. of North Dakota 


(Fed. Projects) 
Title I- III 


Morris, Earl W. 

103 'S. Washington St. 
Carbondalc, 111 
Regional 7,ab 


(Program Evaluation) 
School Programs ' 


Ncarine, Robert J. 

Board of Education 
249 High Street 
Hartford, Conn, 06103 
(Hartford Public Schools) 


(Progra to E v a 1 u a t i o n ) 
Federal Projects 


Me i man, Albert M. 
Ofs. of Supt. 

Bucks County School 
110-A Chapman Lane 
Doyles town. Fa. 


(Program Evaluation) 


Nelson, Dr. A.K. 

Natl. Old Line Bldg - SCREL 
Little Reck, Ark. 

(Regional Lab) 


(Regional Lab) 
Program Evaluation 


Olson, I,o Roy A . 

Office of Evaluation Services 
Michigan State Univ. 

East lansing, Michigan 


(Institutional Evaluation) 
Coll, Evaluation 



162 



(Fed. Projects) 

Title XII Evaluation 



01 sow, Nori.mte K. 

Ninth DislticL Eel. Services 
Cent. vi. 

P.O. box 1075 

Gaii K'svilk, Georgia 30501 



Peterson, Dwain F. (Institutional Evaluation) 

P.O. Box 75 Mankato State College ' Institutional Evaluation 
Mankato, Minn. 56001 



Preising, Paul P. (Program Evaluation) 

632 St. Claire 

Palo Alto, California 

(Stanford) 



Pugh, Richard C. (Institutional Evaluation) 

401 Enstside Drive 
Blooming tot:, Indiana 47401 
(Indiana Univ. 



Rowe, G. Dwight 
Milwaukee Public Schools 
5225 VI.* Vliet Street 
Milwaukee, Wis. 



Puzzuoli, David Anthony (Program Evaluation) 

Ed. Research &. Field Services 
West Virgin i r Univ . 

Ogle bay Hall Annex 
Morgantown W. Va. 26501 



Rossmann, Jack E. ' (Institutional Evaluation) 

465 Vermont Ave . 

Berkeley, Calif. 94707 
(U.C.B.) 



Rowe (Federal Projects) 

Title 1,111 



Satterfield, Christina He it linger (Federal Projects) 

project Reachigh Title I- III 

Route 7/6 

Clarksville, Tenn. 

(Peabody College) 



Schultz a Kenneth M. (Program Evaluation) 

466 West 41 Place 
Hialeah, Florida 33012 
(Dade County) 

ERIC 






(To ac her s Assoc.) 



Shea, Jnr L . 

1549 Wyoming Ave. 
Schenectady 9, W.Y. 12309 
Teachers Assoc. (N.Y.) 


(Teacners Assoc.) 


Shedd, A. Neal 

400 Maryland Ave. S.W. 

Rm 2177 

Washington, D.C. 20202 


(Fed. Projects) 


Sheverbush, Robert L. 
1115 N. El Paso St. 
Colo, Springs, Colo. 


(Program Evaluation) 

! 


Simmer, Lowell W. 

Elk Grove Training & Dev. 
Center 

1706 Algonquin Rd. 
Arlington Hts. 111. 


(Fed. Projects) 
Title I- III 


Stallings, William M. 
507 E. Daniel 
Champaign, 111. 61820 
Univ. of 111. 


(Institutional Evaluation) 


Stanley, William 11. 
El Centro College 
Main at I.amor 
Dallas, Texas 


(Institutional Evaluation) 


Stordahl, Kalmar E. 
Northern Michigan Univ. 
Kerquetto, Mich. 


(Institutional Evaluation) 



Tobiason, Ray 

109 East Pioneer, Puyallup 
Washi ngton 9837 1 


(Program Evaluation) 
School Evaluation 


Van Etigen, Henry 
School F.duc . 


(Arith. Eval.) 


University of Wisconsin 
Madison, Wis. 


• * 



(Program Evaluation) 



Virgin, A Hu - 1 L E . 

61 Inver loch]/ give! 

Suite 2 

Thornhill, Ontario 
(North York Board of Ed.) 



Wagoner, Ralph Howard 
Drake University 
College of Education 
26th & University 
DesMoines, la. 


(Teacher Effect) 


Wallace, Gaylc-n R. 

9857 Telegraph Rd. } Apt 8 
La nh am, Maryland 

U.S.O.E. 


(Title I- III) 
Federal Projects 


Ward. Beatrice A. 
Project ED INN 
Box EDT.NN 

Monterey, Calif. 93940 


(Fed. Projects) 
Title I- III 


Warmbrod, J. Robert 
Center Voc. Tech Ed. 
The Ohio State Univ. 
980 Kirmear Rd. 
Columbus, Ohio 432 12 


(Vocational - Tech.) 


Whiteside, Ray 
ACC Station, Box 576 
Abilene, Texas 79601 
Abilene Christian College 


(Institutional Evaluation) 


Wigderson, Harry I. 

1500 So. Mooney Blvd. 
Visalia, Calif. 93277 

Williams , Lois Q. 

1300 Westview 

East Lansing, Mich. 48823 


(Fed Projects) 
Title I- III 


Williams, Wilbur A. 

Dept. Educ. 

East rn Michigan Univ. 

Ypsilanti, Michigan 48197 

• 


(Institutional Evaluation) 


Wilson , James U. 

Rochester Institute of Tech. 
Rochester, New York 14608 


(Institutional Evaluation) 



Wrigh tsnie.n , Shirley 
Of f i c e l) n i.v . Re s e a i c h 
Box 820 Fisk Uni v , 
Nashville, Term. 37203 

Yarborough, Betty II. 
Chesapeake Public Schools 
Adminis tration Building 
Chesapeake, Virginia 



Institutional Evaluation) 



(Fed, Projects) 
Title I-J.II 



DESCRIPTION OF PARTICIPANTS AND THEIR NEEDS 



Of the completed application blanks and the evaluation report form 
the following con be reported; 

I . Pret onWnsti^tj^n ^ on p; f JParti; cip ant s 
28 Universities (Private or Public) 

26 School Districts (City or County) 

8 Governmental Organizations 
6 Title III Centers 
5 Regional Laboratories 
• Colleges 

U State Departments of Education 
2 Professional Organizations ( * J • E « A « ) 



1 Accreditation Organization 



1 Research Laboratory 



1 State Superintendent of Public Institution 



IT. Kinds of 



j o c l g_ Par bicip ants were Interested 



16 Titi.e I, 111 

$ Impact of colleges on students 

l\ School programs 

2 Teaching effectiveness - 

Te a cha r boh&vi or 
Kadi c a 1 e d u o s ti on 
Gif tod 

Individual 3 y prescribed instruction 

Impact of the East-West Center 

Federal programs 

Arithmetic 

High School 3 

Urban education 



III. Participants Needs were Vari ed - for example: 

21 Knowledge of evaluation 

9 Knowledge of evaluation design 

7 Techniques for designing instruments 

6 Design models for evaluation 

£ Knowledge of evaluation models 

h Statistical procedures 

3 Application of evaluation principles to a 
specific project 

3 Sampling and scaling techniques 
2 Attitude measurement 

2 Skills in developing instruments that get at 
affective domain 

- Use of pictures, slides for evaluation purposes 

- How to make evaluation more than a number gams 

- Ability to sea what is to be evaluated 

- Problem of consumption of feedback data 

- A researcher handbook 

- Criteria for selection of instruments 

- Evaluation of long range objectives 

- Skills for evaluating effectiveness of college 

programs 

- Methods for dealing with non- quantitative data 

- Knowledge of what * s new in computer technology 

- Research techniques and treatment of data 

- Application of CIPP model 

- Creative approaches to evaluation 



Participants Needs were Varied (con*t) 

- Computer use in multivariate analysis 

- Responsibility of personnel in a public school 

evaluation 

- Good examples that can be graphically and 
. pictorially disseminated 



Constructing decision models 
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TENTATIVE PRESESSION SCHEDULE 
February 3-7, 1968 







Sat. 1:30 - 2:00 

Feb. 3 2:00 - 2:45 

2:45 - 3:00 

3:00 - 4:00 

4:00 - 3:00 

5:00 - 7:00 



Introductions , plans , etc. 

Tyler-report on participants interests and problems. 
Intermission 

Pace-speech "Evaluation Perspectives: "68" 

Individual and group discussions among staff and 

participants 

Social hour. 



Sun. 
Feb. 4 



9:30 - 10:30 
10:30 - 10:45 
10:45 - 12:15 



1:30 - 4:00 



4:00 - 3:00 



Staff panel Alkin, Skager, Stufflebeam, and 

Intermission Pace presentation of various 

Staff panel continued views and concepts about 

evaluation: for example — 

formative, summative, product testing, quality control, 
etc. 

Presentation of the concepts, methods, and instruments 
being developed at the UCLA R & D Center for the Study 
of Evaluation of Instructional Programs. 

Pace, Alkin, Skager 

Individual and group discussion among staff and parti- 
cipants. 



Mon. 
Feb. 3 



9:00 - 10:30 

10:30 - 10:45 
10:45 - 12:00 
1:30 - 2:30 
2:30 - 2:43 
2:45 - 4:00 

4:00 - 3:00 



Stufflebeam-presentation of the concepts, methods, etc. 
being developed at the Evaluation Center, Ohio State U. 
Intermission 

Alkin-cost benefits and systems approaches to evaluation. 
Skager-measurement and design problems in evaluation. 
Intermiss ion 

Strodtback-sociological and societal problems in evalu- 
ation. 

Individual and group discussion among staff and parti- 
c ipants . 



Tues. 
Feb. 6 



9:00 - noon 



1:30 - 4:00 
4:00 - 3:00 



Interest groups, air tentative thoughts about possible 
interest groups are as follows: 

Strodtbaek - culturally deprived 
Skager - testing and measurement 
Stuff lebeam - evaluation processes and procedures 
Alkin - systems approaches to evaluation 
Pace - Evaluation problems in higher education 
Alkin - presentation of a simulated evaluation problem. 
Individual and group discussion among staff and parti- 
cipants. 



Wed. 9:00 - noon 

Feb. 7 

1:30 - 2:30 



To be arranged, except that a portion of the time will be 
reserved for participant evaluation of the presession. 
Pace - summary, priorities for the advancement of 
evaluation. 




REVISED SCHEDULE OF ACTIVITIES 



Saturday, 
1:00 
2:00 
2s 30 
3:30 
$:00 



February 3rd 

- 1:U5> Introduction - plans, etc. 

- 2:10 Tyler - Report on Participants’ Interests 

- 3*30 Pace - Lecture on Perspectives 

- Skager - Discussion of Interests Groups 

-7:00 Social Hour . 



Sunday, February l*th 

A.M. Staff Panel - Questions on Evaluation 

Stufflebeam, Skager, Pace, Tyler 
What is Evaluation? 

Criteria for Evaluation? 

$ 

. Evaluation in Higher Education - Pace 

p.M. Alkin - UCLA Models of Evaluation 

Interests Groups 

Stufflebeaiii - CIPP 

Skager - Testing and Research Design 
in Evaluation 

Pace - Higher Education 

Alkin - Cost Benefit Analysis and 
Organization Variables 
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Monday, February 5th 

A.M* Skager - General Considerations of Data Collection 

Emphasizing Testing Problems 

Stufflebeam « CIPP Model 
P.M. Alkin - Simulation Problem 

i 

Tuesday, February 6th 

A.M. Strodtbeck - Cultural Deprivation 

Skager - Measurement of Change 

P.M. Simulation Problem - Final Reports 

Tyler - Behavioral Objectives 
Alkin - Cost Benefit Analysis 

Wednesday, February 7th 

A.M. Skager - Item Sampling Methodology 

Special Interests Groups 
Pace - Higher Education 
Skager - Change Sensitive Test 
Stufflebeam - Evaluation 
Alkin - Systems Analysis 
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PRESESSION MATERIALS 



1. Report and Analysis of an Evaluation Study 

2. Form for Re-analysis of Your Evaluation 

3. The Use and Abuse of Evaluation in Title III By Daniel S. Stuff lebeam 

4. Course Improvement Thorough Evaluation By Lee J. Cronbach 

5. The Countenance of Educational Evaluation By Robert E. Stake 

6. AERA Monograph Series on Curriculum Evaluation: Perspectives 
of Curriculum Evaluation (1) By Ralph Tyler, Robert Gagne, 

Michael Striven 

7. Tentative Pre-session Schedule 

8. Evaluation Perspective: 68 By C. Robert Pace 

9. Letter of Acceptance with Hotel Reservation Card enclosed 

10. AERA 1968 Research Training Pressions Program Participant Evaluation Form 

11. Presession Applicant List 1968 
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PRESESSION CRITIQUE FOR STAFF MEMBERS 



Indicate your observation arc judgment by checking each i tern in one 
column at the left. Items net applicable or not subject to your ob- 
servation should be emitted. Se frank. 



I . 



2 . 



5 . 



Environmental conditions 
a. Classroom spaces 
Work spaces 
Living quarte r s 

Teaching equipment, aids (chalk boards, public acc-ress system, etc.) 
Resource material, library 
Eating faci I i ties 



b. 

c. 

d. 

e. 



Pa rt i c i pants 

a. Appropriateness of academic .backgrounds 
Sufficiency of research experience 
Willi ngness to work 
Intellectual curiosity . 

Concern for asp I icab i ! i ty of techniques 
Aspi ration 

Immediate preparation for Presessicn 



b. 

c. 

d. 

e. 

f. 

9 - 



Organization 

a. Adequacy of notice to prospective applicants 
Sufficiency of preplanning 
Smoothness of operation 
Adaptability to obstacles and feedback 
Sensitivity io grievances 
Adequacy of financial support 



b. 

c. 

d. 

e. 

f . 



Schedu I e 



a. 

b. 

c. 

d. 

e. 

f. 

9 - 

h. 

i . 
J- 



Appropriateness of five days for the job 
Time spent effeientiy 
Evenls sequenced appropriately 
Punctuality 

Balance between formal, informal affairs 

Quantity of d : scuss ions 

Quality of discussions 

Quality of fo r "=l presentations 

Uncbtrus i veness of evaluation efforts 

Methods of evaluation 



Outcomes 

a. Intended center.* was actually taught 
Increase in participant understand i nc 
Improvement i- attitude toward research 
Personal associations initiated 



b. 

c. 

d. 






J 
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6. In general was the Presession well organized? 

yes: 2 no: 2 [because of changes in plans] 

7. Were the facilities suitable for the activities which you had planned? If not, 
specify. 

no: 6 [too small, too hot; not enough small group space; (1) needed better 

' audio-visual equipment)] , 

. | 

8. Should Pressions be limited to the same hotel, or the same city, in which the 
annual meetings will be held? 

same hotel: no: 1 yes: 3 same city: yes: 1 no: 2 

9. Were you to do the same assignment over, in what major ways, if any, would 
you change your contribution? 

individual answers 

10. Do you wish that the Director had made, firmer arrangements to assure 
participants and you of the staff opportunity to meet in pairs or small groups? 

2 as much as possible was done 
1 wa s done 

1 yes 

2 not necessary 

11. Were the objectives you set for yourself during the Presession attained? 

yes: 5 no: 1 

12. Are you inclined to urge your colleagues to become s t d members for such 
an institute or Presession? 

y es; 5 no: 1 [unless there is adequate preplanning] 

13. In what ways, if any, did you as a staff member benefit personally as a 
result of your participation in this Presession? 



% 



individual 



j.75 



) 



PARTICIPANT evaluation form 



l.a. To what extent did the relative availability or vuiaval2.abi3.ity of Loo.-a-,. 

journals interfere with or promote your attaints to master the content or 
this session? 

General consensus was that the unavailability of books and journals 
did qot interfere with attempts to master the content of the session. 

FREQUENCY COUNT j 

i 

None or very little 25 

Not applicable 9 

No response 6 

One person would have lilted to have reference bocks available to 
browse , Two persons stated that no b o o Its were av a I ab Lc and t ha 

_ . - i - 1 ~ 

Weld Liitu umy icopuuot. 

b. To what extent did reproduced materials given to you by the staff improve 
matters? 

General concensus seemed to be that what was available was helpful 
but that more material was needed. The most extensive consent made 
derails the ambivalence felt by most of the respondents: "Stuf f lebeam’s 
paper was excellent. Simulation materials he lpfu 1- - except complete 
sets not available except to 'group chairmen.' Majority of presenta- 
tion**, expecially on UCLA R & D Center activities should have been 
reproduced and distributed rather than read-- tremendous waste of time. 

Also, group reports could have easily been reproduced. Finally, several 
participants expressed interest in acquiring p. 1 of "Assignment 1" 
from fellow participants-** this could have easily been produced in adv a ne e 
by the staff. The lecture method was time- consuming- -bulk of lectures 
could have been reproduced." 



FREQUENCY COUNT 



\€Ty he 


..15 


Moderate ly 


.. 4 


Some w ha t 


. .13 


Very little 


.. 8 


None vs s available 


.. 1 


No Response 


.. 2 


2. a. Did you feel 
grci »ps? 


that you 


FREQUENCY COUNT . 


Adequate 


...35 


Inadequate 


... 8 



lacked a "place to work," either alone or in snail 



Elaborations on the responses stating that the facilities were inadequate 
included the following:* To small and too poorly ventilated; didn't, krov 
v?hat part of the facilities was supposed to be used at any given tire. 




• n 



176 



b. Vlas your rcan satisfactory? 



Gsr.tr a i response was 
given the following 
but correct ions were 



"yec" without qualification. When a 
criticisr3 wore made: Too small as a 
made; too crowded; too hot. 



'‘no" respons 
meeting room; 



bad 



FREQUENCY COUirf 



Yes.. 27 

No 16 



3. a. 



Which features of the meeting rooms were f inadequate or not 
learning? 



coriclucj ve 



to 



FREQUENCY COUNT 



Shape of room v?rong 2 

Size of the room too small for group 31 

Ventilation b ad, too hot and/or too cold, 

and too stuffy..... 21 

Too noisy.,, 3 

Not adequately equipped for the type 

of meeting 4 

1) lighting bad 



2) No place for note taking, no table 



space 

3) Not enough chairs 

No Response 4 

Irrelevant Response.... 1 



b. Which features were especially facility cive in the same regard? 
FREQUENCY COUNT 



No Response ' 21 

Another point in time was better, 

the room was re-arran^ed or 

changed to one more adequate . 8 

Sense of corniunity 1 

Good Lighting 3 

Rooms all in same hotel 1 

FA system available 1 

Chalkboard available 1 

Comfortable seats 1 

Smaller groups 1 

None . . . 6 



e 



AERA Presession Evaluation 
Scheduling and Organization 



There were 43 questionnaires completed but many individuals did not 
respond to every ?,ter... Many of the items, therefore, do not have 
43 responses. The nw-.ber of responses are reported .rather than 
percentages of the total. No assumption? are made regarding non 
responses. 



177 



4 . a. V.V.l? five d:y3 too lcr;j a period to leave your work at hare for the purpose 
of attending till r, ronsion? 



a* Flv" day s away fro:.i their vork was not too long for 29 particl- 
^ts but it vas too lon 3 for 11. A few individuals indicated that 
3 or 4 days would have been adequate. 

b. 'Was five days tco short a period in which to learn much of the content of 
this session? 

b 34 participants felt that 5 days vas not tod short a ticse to 
leavu the content of the session. Only two participants felt it 

to be too chor t . 

5 . a. Were you allied enough time in which to pursue activities of your own 

choosing? 

5a Host respondents felt that enough tine was allowed for activities 
of ’their own choosing (31) bat 9 felt there was not enoug.. 

b. Would you have preferred not to meet in the evening after dinner? 

I 

b There was much confusion about evening meetings and 22 indicated 
that tb"' question was not relevant since there were no evening 
s^tin^. Eight people indicated that they would like to have met 
• in thokvenings and four said they would not want to have evening 

tr.ee tings. , , . 

e Would vou have preferred more or fewer meetings per day than there actua .y 
wSe? S v^s the number of meetings per day agreeable to you? 

c. 28 persons said that the number of meetings per day was O.K. 

Responses to the first part of the question were confusing since . 
a yes or no did not qualify whether more or fewer meetings p~ 

day vert* preferred. 

6 . a. V.’are the irdivio.ual lectures too long to sit and listen or take notes? 

6a 26 persons indicated that the lectures vere not too long, 9 
indicated they vere too long, and 5 indicated they vere sometimes 
too long. The suggestion v?as made that a variety of media vou Id 

have been bettor . 

b. Were the lectures scheduled in an appropriate sequence? 

I 

b. Personal opinion vas split 14 to 14 on the appropriate sequencing 
of tho lectures. Four others said it vas as appropriate as it could 
be end 5 persons said it vas unimportant. 

7 . Did you have sufficient opportunities to internet with other participants? 

7a. Thair ty- two participants had sufficient opportunity to iiitciact 
with othars and ccvon said there vas not enough opportunity. It vas 
suggested that “an activity to break tn-* ice* working in small groups 
the first morning, would have enhanced the interaction." 



a. Were U.? instructors too i ncccessible or unapproachable so that you did 
not. c:.t the ir.dJ.vidu r il attention that via desired? 

8n Thircy-fivi parsons fc:lt the instructors were approachable and 
accessible. Only six persons felt they cere not accessible. 

VJus it helpful to have graduate students assistants present? 

b. Host people wave Amused at thi question. One respondent boldly 
'said tnore ware '10 graduate students present and six as hed if there 
\;ere any present. Season* remarked that he wasn’t utilized enough 
aatfd several indicated that u;ay have helped the staff but he didn't 
help th.-m. Five people said he wasn’t helpful and the majority of 
people were quite indifferent about tU- question. 

Did the- attempts to evaluate your progress and reactions during the session 
(and at this rf.ov.enL) interfere with ycur v.ork here? 

9a. Thirty- four people indicated that the evaluation did not inter- 
fere with their work. Co:, '.rents like the- following were also nude. 

"Did not like repeating assessment ." "None Trade." "This one is 
fine but the several page one was ridicu i our> . " "Would have preferred 
more on-going evaluation." 

In general, was the Presession well organized? 

10a Twenty- two participants felt that the Session was riot well organized 
and eleven thought it was well organized. Several cor.v.onts were made 
about the week beginning and tlw changes that were always being made. 

Item 4a and 10 were brought togethei in the following statements. 
Regarding five days away from work: — 

"Not if preparation for this presession had been effective. One 
day was lost in organizing. This could have been reduced, to ,30 
minutes with adequate pre-planning." 

From the same point of view; — 

"No, although real content of th is session could easily have been 
covered in three days if "lecture" materials had been printed and 
distributed." 



AERA Presession Evaluation 
Content and Presentation 



a. Did the content of the lectures arc! readings presuppose far more previous 
training than you had? 

11a. Host respondents (36) felt that the content of lectures and 
reading." did not presuppose more training than they had previously. 

Five respondents believed that their previous training had not 
adequately prepar’d them for the lectures and readings. 
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l].b. Should less in these areas or more have been presupposed? 



b. Kost rcfr P ond?nt« (24) thought that the level presupposed train- 
ing was adequate. Seven felt that more training should have been 

presupposed; two felt that Jess training should have been presupposed: 
and four didn't knew. f rf ’ 



12 . 



To wheit extent V.JS the content of the lectures and readings relevant to 
what you hoped to accomplish during the! session? 



12a. The re. avarice of the content of the lecture s and readings v;as 
rated very relevant by 19 respondents, moderate 1- relevant by 13 
respondents, and not relevant by five respondents. 

13. a. Were tie lectures stimulating and interesting? 

13&. Thu let -orea were thought to be stimulating and interesting by 
26 respondents , or iy KOderately stimulating by 11 respondents, arid 
not stimulating ot interesting by 2 respondents. 



13. b. Were the lecturers competent to speak on the subject assigned them? 



b. The competence of the lecturers was clearly acknowledged by 39 
respondents and questioned by 2 respondents. 



13.c. Were the 2>.*ctur:ers well prepared? 

c. The lecture" were considered well prepared tv 23 respondents, only 

- 1 11 1 1 . « m « « « 
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moderately well prepared by 8 respondents, and not at all veil pre- 
pared by 6 respondents . Skager's presentation was ireruioned as a 
particularly well prepared lecture. 
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Were you disappointed in any way with the group of p-ir ticipants? 

14a. Host respondents (28) were not disappointed with the group of 
participants, 6 were moderately disappointed, and 3 were quite diem 
ppointed. Arong the latter two categories, the typical corr.r.ent vus 
tnat the group was far too diverse. One respondent suggested that 
a smaller, more heterogeneous group should hav* been selected. 



If you had it to do over again would' you a H ;iv for this Pre«-=sion 
you have just canpleted? re -“ bolcn 



15e. V/hile 23 participants indicated that they would apply for tK- 

Presession if they had it to do over again, 13 participants Indicated 
that they would not. 



16 ' lit ^Xt^tSndT 8 ' 18 h ° ld '" DUld — ** to others 



16a. Yes *•- 27 



Ko « 12 



Don't know « 2 



17. 



.f 3 i ? ipat f ^. inta ’ ;nil> 3 **« sort of contact with at 
the Profession staff? , 

J 2a . y.or, m 35 



«ast one o: 



Ko « 6 
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contribution to tduo-,1 



18a. Yes »- 36 



No «* 3 



19. 



“5 ™ sssy - 

N° * 33 Don ’ t know » 3 



19a. Yes « 5 



this PresoFs^n 3 /•->>*! W i^ co ^ a ^ ; ^te m research with sornr no eis 

.o .,ier than those you already were likely fc : ' 

No « 20 !)o;;'t know « 2 



20*. Yes *= 16 



21. Do you fool that the staff should feel chat it has accompli., *i*. h* 
during # thif five-day presession? 

2l£. Yes « 25 Ho » 14 Don't know = 2 

r'hd,; T r ?! po :dd " ye£ " to ehx# u ** wciiy hS d no forth*, c OC; „, 

f ; 10 negatively ofter. had soxe additional thoughts c .i-- 

h y®P° ndod . "datively felt that the objectives had not bee, 
*rly stated hsnee, they could not decide whether the staff had 

cor.r.entf : ltS ° bjectlves ' Xhe followin S ware typical of the negative 

the stace of the srt of 

...I felt frustrated and bored waiting for the staff to organize 
its material ... this v?as not my first presession." 

""tir^ro^S^ 1 ? 3 WaSte f, ln ir " cha P ics of or 8anizing the presession 
•••Cliw. robbed from small group participation time." 

Positive consnsnts included: 

* 

"...Dr. Pace and others did a fine job." 

"...Evaluation precession staff v?ere great." 



ion by 



kc it nore 



; attending 
rate with)? 



objectives ; 
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EVALUATION OF PARTICIPANTS STUDIES 



Before coming to the UCLA Pre-Session on Evaluation, par- 
ticipants were asked to respond to a number of questions about 
evaluation studies in which they were actively engaged. Specif- 
ically, these questions were: (1) "Describe a program you are 

now evaluating (or have some part in evaluating) its nature, 
content, scope, purposes, etc." (2) "what sorts of data (informa- 
tion) are you collecting (have collected) and how did they relate 
to the evaluation purposes?" (3) "What problems, difficulties, 
etc., (procedural, Organizational, analytical) are you encounter- 
ing and how are you dealing with them?" (4) "What sorts of con- 
clusions are you drawing, or hope to draw, from your evaluation? 
And what benefits (changes, decisions, etc.) and for or by whom, 
do you hope will result?" (5) "What skills, knowledge, etc., about 
evaluation do you think you need in order to carry out your role 
more effectively?" 



Participants who had failed to respond to this request 
before the pre-session began were urged to do so at the first 
meeting. The staff was thus able to obtain the information from 
all but about ten of the eighty-seven participants. It was ob- 
vious that all or nearly all of the participants were active in 
some sort of evaluation research. 



At the close of the pre-session, participants were asked 
to answer the same questions a second time, but this time indicat- 
ing any changes in the conception or design of their evaluation 
study traceable to the pre-session experience. Forty-nine of the 
participants responded to this request. These responses, dis- 
cursive in form, were enalvzed for instances in which topics 
presented during a pre-session were mentioned. The pre-session 
topics are listed below in order of the frequency they were cited. 
The number in parenthesis following each topic indicates the number 

of such citations. ! 

1 

INCORPORATION OF PROCESS OR FEEDBACK DATA (25): References, in 

one form or another, to the need for some sort of monitoring of 
developmental phases of instructional programs. In the main, 
these comments referred specifically to the "process" aspect of 
the CIPP model presented by Stufflebeam. 

j 

ITEM S AMPLING (24): This topic refers to methods of data collec- 

tion proposed by Frederic Lord and designed to permit simultaneous 
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sampling of subjects and test items. Item sampling methods have 
particular appeal in evaluation research by providing for the 
coverage of a wide domain of test content coupled with a highly 
economical expenditure of examinee time. 

BEHAVIORAL OBJECTIVES (13) : An indication of the need for 

specifying instructional objectives in behavioral terms as a part 
of evaluation research. 

ANALYSIS OF CHANGE SCORES (12) : An indication of increased 

awareness of the importance of using appropriate methods of data 
analysis when studying change in individual performance in the 
evaluation of programs of instruction. 

CONSTRUCTION OF "RELEVAN T" TESTS (8): This category includes ex- 

plicit recognition of the limitations of standardized achievement 
tests in the eva3_uation of instruction and increased knowledge as 
to procedures for constructing local tests. 

COST EFFECTIVENESS (6): Recognition of the need to interpret 

evaluation data in the light of economic considerations • 

CONTEXTUAL VARIABLES (6) : Recognition that instructional outcomes 

must be interpreted in terms of societal, organizational, and 
other forces that moderate their effects. 

LIMITATIONS OF "T RUE" EXPERIMENTA3j DESIGNS (5) : Indications^that 

the requirements of rigorous laboratory research often cannot be 
met under conditions imposed on the typical local evaluation 

study . 

MISCELLANEOUS: Occasional references were made to additional 

topics dis cussed during the pre-session. These included references 
to the "hidden curriculum" (Strodbeck) , profile analysis, measuring 
student and teacher attitudes, relevance of simulation methods in 
evaluation, the use of time logs of activities, and the desire to 
give a feedback from the pre-session to members of local staffs. 

The topics mentioned above provide an ordering with respect 
to perceived importance of the conceptual and methodological con- 
cerns of evaluation researchers attending the pre-session. As far 
as the information derived from these reconsiderations of local 
evaluation studies was concerned, there were very few references to 
topics that were not included in the pre-session. One participant 
mentioned a need for knowledge of computer techniques, and three 
individuals suggested that they needed greater knowledge of statis- 
tics. While these data indicate •saJko * the relative importance of 
the various topics included in the pre-session, they do not give us 
a great deal of information about topics that should be included m 
future training sessions. 



R.W.S. 
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UNOBSTRUSIVE MEASURES 



The staff members have had many lettex from participants and 
from individuals who heard about the pre-session. 



Following are some of the requests: 

Skager - Requests for reprints 

request for exchanging data 
requests for consul ting 



Alkln - requests for sharing articles 

requests for simulation materials 
U.S.O.E. (Dr. Shedd) wishes our R & D center to 
further develop simulation materials 
so it can be used to train Title III 
project directors 



Tyler 



request for planning add teaching workshop for 

teachers in State of Connecticut, (Burke) 
request for material on objectives 



Face - request for speech 

request for all. materials used 
many letters of thanks 



DIRECTOR’S NARRATIVE REPORT 



In proposing and planning the presession, our general orientation 
was, I think, something like the following: evaluation is widespread; 

evaluation is in a state of flux; a great variety of activities are called 
evaluation with little commonality in vocabulary or concepts; many of us in 
the R & D Center at UCLA had given a good deal of thought to ways in which 
the concept and practice of evaluation might be improved; an exchange of 
viewpoints and an effort to clarify and expand major problems and needed new 
directions should be generally useful for the field. 

We selected participants who were responsible for evaluation programs 
or who, in other ways, were facing practical evaluation problems. 

During the first day and a half, we sensed that our presentations 
were too theoretical. We had presented, and tried to generate discussions of, 
such matters as the difference between formative and summative evaluation, 
between process and produce, the role of the evaluator, the CIPP model for 
evaluation, the meaning of "contextual” variables, etc. For the last three 
days we shifted to an emphasis that was, alternately, more practical and more 
technical. We had developed a "simulation” problem for the presession, and 
this proved, I think, to be quite successful. It should have been scheduled 
earlier in the program, partly because it helped to illuminate some of the 
sticky problems evaluators face, and partly because it got people working on 
and talking about a common task. We also had small groups meet to discuss 
special problems — people doing evaluation in higher education, for example. 

Had space been available we would have had more small group sessions, for in 
these smaller groups the participants had a better chance to raise questions 
and discuss possible solutions. We also had some straightforward presentations 
of measurement and methodological topics — such as criterion-based tests, item 
sampling, change scores, and change-sensitive tests. These presentations were, 

I believe, informative and helpful. 

By the end of the five days, we had done most of what we had intended 
to do. If we were doing it again, I think we would spend more time on simulated 
problems, more time on technical problems, less time talking about concepts and 
points of view, provide more time for individual conferences with staff 
members, and more time for special interest groups. 

I suspect that any presession on a topic as general as "evaluation” 
will attract a heterogeneous group of participants — at least more so than some 
well defined technical topic would attract. I believe that some AERA pre- 
sessions should be organized around these more general problem areas, but per- 
haps one sl.:>uld think of them as symposia or workshops rather than as a train- 
ing program. 



THE COMPUTER AND NATURAL LANGUAGE 



Director 

Dr. Ellis B. Page 
University of Connecticut 



INTRODUCTION 



Each year prior to the annual meeting, the American Educational 
Research Association sponsors a series of five-day workshops or presessions. • 
Supported through a grant from the United States Office of Education, no 
tuition charges are levied against the participants. This report describes 
one such presession — Presession number 7: The Computer and Natural Language. 

Under the leadership of Ellis B. Page, he, and eight other staff members 
presented various natural language computing strategies to 27 participants. 

The content of the presession ranged from general "state of the art" reviews, 
to specific, and relatively technical, natural language computing applications. 

This brief introduction is followed by a general description of the 
organization of the presession. Specific objectives as well as the general 
content outline will be presented. The unique contributions made by each of 
the nine staff members in the presentation of the content will be cited. And 
a list of reproduced materials distributed to the participants is also included. 

This is followed by a description of the 27 participant ^ of the presession. 
A y '"er a brief consideration of the criteria used by the staff in selecting 
participants^ and after a listing of their names and addresses, several tables 
are presented which serve to describe the general characteristics of the group. 

The fourth part of this report presents some of the results of various 
formative and evaluative instruments which were administered to the partici- 
pants throughout the five days of the presession. Also included are the results 
obtained from a questionnaire asking the staff 1 s reaction to how well the 
objectives of the session had been met. 

In conclusion, this report contains some of the Director’s own reactions 
to the presession and suggests some of the changes which ought to be incorpor- 
ated in future meetings of this kind. 



ORGANIZATION AND CONTENT OF THE PHESESS.ION 



Before we consider the objectives and the content of this presession, 
let us turn to some of the rationale on which the objectives and the selection 
of the content was based. 

Since ancient times, the main line of Western scholarship has centered 
around the analysis, transformation, and production of written language. This 
remains largely true today, in the fields loosely termed the "Humanities", and 
it remains equally true in those educational studies usually regarded as "less 
scientific": philosophy, history, administration, curriculum, and various 

subject-matter teaching fields. Until very recently, such fields have lacked 
the tools and techniques and concepts necessary for large-scale, reliable, and 
inexpensive analysis of language j and high-level, valid generalizations and 
predictions have thus been difficult to make. Largely as a result of these 
restrictions, the traditional fields of verbal scholarship have been left far 
behind during the accelerating advancement of the sciences. 

Recent development in computer science, however, make possible a 
revolution in verbal scholarship, and suggest radical changes in the scope 
and depth of certain kinds of educational research. The softer areas of 
professional education (such as educational history and philosophy) may be 
made much more researchable by the application of computer techniques to 
verbal data. Educational administration may be much increased in rigor by 
the analysis of laws, codes, decisions, board deliberations, etc. Guidance 
may be made much more effective through automatic access to large verbal 
data bases and high-speed retrieval techniques. Perhaps most promising of 
all, the writings of students themselves may become subject to analysis, 
amendment, and evaluation, both in batch-processing and in conversation-mode 
employment of computers . 

Even those behavioral researchers who use computers regularly, however, 
are unlikely to understand their employment with verbal data, or to be familiar 
with the work in artificial intelligence, computational linguistics, and other 
fields, which has already gone on. This five-day presession, therefore, aims 
at introducing able and active researchers, and research teachers, to the con- 
cepts, skills, problems, and accomplishments of natural-language analysis. 

The eventual outcome expected is that they will do research using natural- 
language analysis, will be able to help researchers in the softer areas perform 
such analysis, and will teach such research techniques to students and faculty 
in their own regions and institutions. In this way, an effective and virtually 
new area of educational research will be opened for active exploration and 
employment . 

Primary among all of the things which we tried to accomplish at the pre- 
session were the following four basic objectives. These objectives sought for 
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the participants were as follows: 

(1) To learn about list processing languages, and. to master actual 
programming techniques, which are especially suited for natural ' 
language programming. 

(2) To learn strategies of heuristic programming, psychological 
simulation, sematic memory, and artificial intelligence appro- 
priate for such analysis. 

(3) To learn techniques of essay grading by computer, content analy- 
sis, and computational linguistics which may illuminate certain 
educational problems. 

(4) To conceive, flow-chart, and partially program some suitable small 
strategies for possible later incorporation into larger educational 
researches. 

I 

' The following content outline and calendar for the presession indicates 
in greater detail what topics were covered during the five-day workshop. 
Obviously, not all of these topics could be covered in great depth. However, 
the participants were, at least, exposed to all of these items, and frequently 
many informal discussions with individual staff members allowed the participants 
to delve much more deeply into many of these areas than was possible during the 
regularly scheduled sessions. The names in parentheses following each topic on 
the outline indicate what staff member had the major responsibility for the 
presentation of the materials associated with each topic. Note that we say 
S&.iffr responsibility and not sole responsibility. All staff members were 
expected to, and frequently did, contribute to the presentation and discussion 
of materials. 

One item which does not appear on this outline in any detail is the 
small group discussion sections which took place almost every evening of the 
presession. Here the participants had the opportunity to select among three 
or four sessions which ranged in scope from elementary deductive logic to 
rather technical aspects of computer programming. Also, a remote IBM QUICKTRAN 
terminal was available for the participants. Many took advantage of this to ' 
learn the rudiments of timesharing and to review basic FORTRAN II programming. 



i 
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CONTENT OUTLINE AND CALENDAR FOR PRESESSION 

"The Computer and Natural Language" 

' 1 ’ 

FRIDAY NIGHT : Arrive at Pheasant Run Lodge, St. Charles, Illinois. 

Staff: Planning and Preparation 

Participants: Informal social period for those present 

8:00 - 11:00 P.M. 

SATURDAY : Coffee: 8:30 A.M. - 9:00 A.M. 1 

9:00 - 12:00 A.M. 

I. INTRODUCTION. Background of the participants. Structure 
of the Seminar-Workshop. Overview of the field. (E. 

( 

II. NATURAL LANGUAGE PROGRAMMING. — All present staff. 

A. Principles and conveniences: 

1. Alphameric arrays (E. Page) 

2. Table look-up strategies (S. Sedelow) 

3. Strii.g manipulation (D. Paulus) 

4. List-processing (G. Fisher) 

5. Dynamic storage (G. Fisher) 

‘ 1:00 - 3:00 P.M . II continued: 

B. Using FORTRAN for natural- language programming 

1. User-written subroutines (D. Paulus) 

2. FORTRAN-imbedded conveniences such as SLIP, 

DYSTAL, STUFF (D. Paulus) 

C. Alternative languages: LISP, SNOBOL, and 

especially PL/I. (G. Fisher and S. Sedelow) 

[The above will be an overview. These topics will be 
studied also in voluntary skill sessions throughout 
the workshop, and developed further from time to time.) 

3:30 - 5:00 P.M. 



III. INTERDISCIPLINARY STRUCTURE OF THE WORK 

A. Parent societies: LSA, ACM (SIGs & SICs), ADI (ASIS) , 

AMTCL, AFIPS (FJCC, SJCC) , IFIPS, LSA, MLA, APA, 
AERA (W. Sedelow). 

B. Periodicals and publications (W. Sedelow) 

» • 

C. Major centers and projects (W. Sedelow) 

5:00 P.M . 

Reception, dinner 

8:00 P.M . Voluntary workshops or interest groups: 

FORTRAN, PL/I, special problems as determined by 
interest of participants and staff. 



SUNDAY : 
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A.M. Free except for special interest and skill sessions: 



1. PL/I programming (G. Fisher) 

2. Symbolic logic: Propositional and predicate calculus 

(E. Page) 

3. Bookshelf open (true during most free periods) 

1:00 - 4:00 P.M. 



IV. APPLICATIONS TO HUMANITIES and S. Sedelow) 

A. Categories of measurement 

B. Indexes, glossaries, bibliographies, information 

retrieval 

C. Stylistic analysis 

D. Comparisons and trends: authors, periods 

E. Steps toward criticism 

F. Status of the field: . current work and findings, 

relevance for education 

4:00 - 5:30 P.M . Special interest seminars and groups 

Evening : Special interest seminars ^nd groups , 

Work on the computer terminals, etc. 

Bookshelf opei* i 



MONDAY : 

8:30 - 12:00 A.M . 

V. LANGUAGE ANALYSIS 



A. Content analysis, such as General Inquirer (D. Paulus) 

B. Essay grading, such as PEG (E. Page and D. Paulus) 

C. Probabilistic vs. Deterministic strategies (E. Page) 

D. Parsing for syntax and for meaning (E. Page) 

E. Discourse and anaphoric analysis (E. Page) 

F. Information retrieval strategies (C. Helm) 

1:00 - 4:00 P.M . 

VI. SIMULATION OF VERBAL BEHAVIOR (C. Helm) 

A. Verbal learning, such as >EPAM, WEPAM, etc. 

• B. Semantic memory, such as Raphael and Quillian 

C. Decision-making 

D. Measurement problems and scientific strategies 

E. Structure of simulation languages 

F. Formulating languages and data structures from 

problem specification 

4:00 - 5:30 P.M. and Evening : Voluntary workshops or interest 

groups . 

Beginning own construction of proposed research in 
natural language analysis 
Working with computer terminals 
Reading, etc. 
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I HflDSDAY : 

8:30 - 12:00 A.M . 

) VII. ARTIFICIAL INTELLIGENCE 

A. Search (E. Page) 

B. Pattern recognition, and relevance to NLA (E. Page) 

C. Learning systems and the improvement of analysis (E. Page) 

D. Problem-solving and planning (E. Page) . . 

E. Statistical strategies in pattern classification (C, Helm) 

F. Induction and models for future analysis (C. Helm) 

1:00 - 4:00 P.M . 

VIII. SOME EDUCATIONAL APPLICATIONS (A. Ellis) 

A. Turing machines and computers 

B. Interaction in instruction and guidance 

C. Educational languages, such as TRAC, ELIZA, etc. 

i D. Use of information retrieval in guidance: ISVD, etc. 

• i ' ■ * . ■ ' 

4:00 - 5:30 P.M . 

Small groups and individual problems 
Work with the computer terminals 

Evening 

More small group and individual work. Terminal and 
reading, etc. 

WEDNESDAY: 



8:30 - 12:00 A.M . 

IX. FURTHER STRATEGIES IN NATURAL LANGUAGE ANALYSIS (Staff) 

A. Ready-to-use systems and subroutines 

B. Ready- to-use data sets and documents 

C. Automatic dictionaries and encyclopedias 

D. Cooperation among researchers 

E. Additional topics 

1:00 - 2:30 P.M . 

X. SUMMARY AND EVALUATION OF THE PRESESSION (E. Page and D. Paulus 
2:30 - 3:30 P.M . 

Final social hour and leave-taking 

4:00 P.M. Chartered bus leaves for Conrad-Hilton Hotel, Chicago. 

“ 1 (To arrive by around 5:00 P.M.) 

Others leave for Airport. 
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As previously indicated, the staff of the presession consisted of 
nine members. Each staff member was selected because of his or her special 
expertise in the area or topic which they were to present. Not every staff 
member remained at the presession for the entire five days of the meeting. 

In the following list of staff the number in parentheses indicated the 
amount of time spent at the presession. 

(1) Ellis B, Page, the director of the presession (5 days) 

Bureau of Educational Research 

University of Connecticut 
Storrs, Connecticut 

(2) Dieter H. Paulus, the as si mt director of the presession (5 days) 
Bureau of Educational Research 

University of Connecticut 
Storrs, Connecticut 

(3) Allan B. Ellis (3 days) 

New England Educational Data Systems 
Harvard University 
Cambridge, Massachusetts 

(4) Carl Helm (3 days) 

City University of New York 
New York, New York 

(5) Sally Y. Sedelow (3 days) 

Department of Information Science 
University of North Carolina 
Chapel Hill, North Carolina 

(6) Walter Sedelow (3 days) 

Dean of School of Library Science 
University of North Carolina 
Chapel Hill, North Carolina 

(7) Gerald A. Fisher, Jr. (5 days) 

Bureau of Educational Research 
University of Connecticut 
Storrs, Connecticut 

(8) John McManus, assistant to the presession staff (5 days) 

Bureau of Educational Research 

University of Connecticut 
Storrs, Connecticut 

(9) Louise Patros, secretary to the presession staff (5 days) 

Bureau of Educational Research I 

University of Connecticut 
Storrs, Connecticut 



STUDY AND REFERENCE MATERIALS 



The staff of the presession set up a library of Si volumes and 
periodicals relating to natural language computing. These materials were 
available to all participants during most of the presession, and they 
were allowed to check these materials out overnight. 

In addition to these materials, each staff member brought with him 
a selection of materials to supplement his lectures. These materials 
served either as explanatory and more detailed materials, or as outlines 
for the content of particular lectures. The names in parentheses indicate 
the staff member who distributed the materials. 

A list of all materials received by each applicant follows: 

A. Instructional Materials 

1. Harvard Graduate School of Education — Annual Report 1966-67. 

(A. Ellis) P„ 101 Progress report of the Information 
System for Vocational Decisions. 

2. MLA Conference 16. "The Application of Computing Devices to 

Scholarships in the Fields of Language and Literature". (S. 

Sedelow) P. 9. Report of MLA Conference 16 (1965). 

3. Ellis — Bibliography. (A. Ellis) P. 9 

A list of 130 Bibliographic items relating to natural language 
computing. 

4. Biographical Data on Participants of AERA Presession 7. (D.H. Paulus) 

P. 3 Seven tables presenting summary information on participants. 

5. Exercise II-A-3. Complete the Following Program. (D.H. Paulus) 

P. 1 Sample program and assignment of string manipulation program 
written in FORTRAN IV. 

6. Sedelow and Sedelow. A Preface to Computational Stylistics. 

(S. and W. Sedelow) P. 14. Discussion of stylistic analysis. 

7. Ellis and Wetherell. The Computer and Career Decisions. (A. Ellis) 

P, 30. Discusses an information retrieval system for career 
choices . 

8. McManus. Computer Evaluation of Teachers' Reports. The Lesley-Ellis 

Project. (McManus) P. 6. A description of the application of the 
General Inquirer to teachers' reports on students. 

9. Levien and Maron. A Computer System for Inference Execution and 

Data Retrieval. (E.B. Page) P. 7 This paper reports - RAND 
project concerned with the use of computers in the logical analysis 
of large collection of factual data. 

10. Content Analysis - The General Inquirer. (D.H. Paulus) P e 3 Brief 
outline describing structure and function of the General Inquirer 
System. 
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11* Quiz on Context-Free Grammars and. Parsing. (G. Fisher) P. 1 
Five item quiz on context-free grammers and parsing. 

12. Elevr Script. (J. McManus) P. 1 Script illustrating the ELIZA. 

system. 

13. Weizenbaum. ELIZA. - A Computer Program for the Study of Natural 

Lang' age Communication Between Man and Machine. (J. McManus) 

P. 10. i 

14. FI002 FAP Fap subroutine which encodes natural language words. 

15. Paulus. Feedback Problems in Project Essay Grade and "Handout to 

Accompany". (D.H. Paulus) F. 14* Description of an interactive 
essay grading program. 

16. Figures 1 and 2. (S„ Sedelow) P. 2. Two flow charts illustrat- 

ing search strategies. ■ 

17. Page. Grading Essays by Computer: Progress Report. (E.B. Page) 

P. 16 Results and strategies of Project Essay Grade. 

IB. Outline for Session on Inter-Disciplinary Structure of the Work. 

(W. Sedelow) P. 2 Outline for Inter-Disciplinary structure of 
natural language computing. 1 

19. Rosenbaum. A Grammar Base Question-Answering Procedure. (E.B. Page) 
P. 6 A description of a procedure for the automatic retrieval 
of information. 

' ,20.. KWIC Handout. (D.H. Paulus) P. 2 A FORTRAN Program for phrase 
look-up. j* 

21. The LISP Language. (D.H, Paulus) P. 19 A general description of 

the LISP language. 

22. LSHFT (D.H. Paulus) Examples of programs written in MAP for 

packing and unpacking strings. j 

23. Kuno and Oettinger. I. Multiple -Path Syntactic Analyzer. (E.B. 

Page) P. 29 General description of Kuno 1 s parsing program. 

24. Asher. New Frontiers in Research; Applications of the Computer to 

Investigations in Reading. (E.B. Page) P. 14 A description of 

some computer applications to research on reading. 

25. Fisher. A Parsing Program. (G. Fisher) P. 12 Flow-chart and 

PL/I procedures for a parsing program. 

26. IIC. Outline for PL/I. (S. Sedelow) P. 2 Outline for PL/I. 

27. Lawson. PL/I List Processing. (G. Fisher) P. 10 Detailed 

explanation of the list processing facilities of PL/l. 

28. Griffiths and Petrick. On the Relative Efficiencies of Context- 

Free Grammar Recognizers. (E.B. Page) P. 12 A description of 
parsing recognition procedures. 

29. Fisher and Hiller. The SCORTXT System: An Approach to Natural 

Language. (G. Fisher) P. 24 .Describes a programming system 
for the computer analysis of text. 

30. Yale Computer Center. SN0B0L. (D.H. Paulus) P. 24 A description 

of the SN0B0L language. 

31. Page. Statistical and Linguistic Strategies in the Computer Grading 

of Essays. (E.B. Page) P. 13 Describes current strategies and 
results of Project Essay Grade. 

32. Madnick. String Processing Techniques. (D.H. Paulus) P. 5. 

Discusses the internal organization of string processing systems. 

33. STUFF. (D.H. Paulus) P. 14 Description of and directions for 

using the STUFF (String Utility Routines for FORTRAN IV) system. 
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34 . 

35 . 

36 . 



37. 

38. 

39. 

40. 



B. Evaluation Materials 

1. .Anticipated Research Using Natural Language. Open ended Question - 

naire requesting information about participants future research 
plans. 

2. Review of Artificial Intelligence. (E. B. Page) P. 2 Nine review 
questions on artificial intelligence. 

3. Review on Computer Grading of Essays. (E.B. Page) P. 2 Seven 
review questions on Project Essay Grade. 

4. Quiz on Context-Free Grammars. (E.B. Page) P. 4 Four item quiz 
on context-free grammars. 

5. Evaluation Form II— B. (D. H. Paulus) P. 1 Open-ended questionnaire 
administered after the second day, which solicits general cx*iticisms 
and comments about the pre session. 

6. Instrument I-B. How Well Could I Do. (D.H. Paulus) P. 2 Thirty 
item self rating inventory on concepts of natural language comput- 
ing- 

7. Instrument I-C. (D.H. Paulus) P. 2 Thirty- item questionnaire. 
Semantic differential structure. 

3. Quiz-List Processing. (G. Fisher) P. 1 Four item quiz on List 
processing. 

9. Quiz on Natural Language Applications to Education. (E.B. Page) 

P. 1 Twenty item quiz on natural language computing applications 
to education. 

10. PL/I Exercises. (G. Fisher) P. 2 Nineteen item quiz on PL/l. 

11. Quiz on PL/I and string manipulation, Forms A and B. (G. Fisher) 
Twenty-five item quiz on PL/l and string processing. 

12. Presessions Critique for Staff Members (For Directors, Instructors, 
and Assistants) (AERA) P. 2 Thirteen item evaluative question- 
naire for presession staff. 

13. AERA 1968 Research Training Presessions Program Participant 
Evaluation Form. (AERA) P. 2 Twenty-one item participant 
evaluation form on presession. 



Weizenbaum. Symmetric List Processor. (D.H. Paulus) P. 14 
Description and rationale of the SLIP Language. 

Syneta Script. (J. McManus) P. 3 Script illustrating the ELIZA 
system. 

Keyser and Petrick. Syntactic Analysis. (E.B. Page) P. 73. 

A theoretical discussion of Petrick 1 s strategies for syntactic 
analysis. 

Outline for Table Look-Up Procedures. (S. Sedelow) P. 1 Outline 
for table look-up procedures. 1 ; j 

Tentative Calendar for Presession. (E.B. Page) P. 3 Schedule 
for the presession. 

The Tree Building and Searching Procedure. (S. Sedelow) P. 2 
An illustrative PL/l procedure for tree building and searc hin g. 

Which Script Please. (J. McManus) P. 3 Sample script for the 
ELIZA system. 




THE PARTICTPA NTS 



.. f 1 ? section, first, the criteria which were used in selecting 

he participants for the presession will he discussed. This will he followed 
y an alphabetical list of the names and addresses of the participants. Then 
some summary data about the participants will be presented. 

+ , . the .announcement and the proposal for the presession discussed 

the type of applicant who would be ideal for this sort of presession. It was 
indicated that this presession, was not intended to be a first workshop in 
P r °| rajm ^ statistics educational psychology, or liberal arts. 

0^+^ WaS hlghly _^ es f rable that participants have some prior competence 
at-,* f th ^ S ? a ^ e J s * The xdeal Participant, therefore, would be an educ- 

o^FOR™w yC w?+h g f T SUr T nt s ? ecialis t, with a fair working knowledge 
of FORTRAN, with some background and interest in the liberal arts, and with a 

demonstrated research drive. He would usually have a doctoral degree but 
some may be pre-doctoral graduate students with unusual promise. 

. . Formal requirements for admission were stated as follows: The parti- 

cipants should have (1) good background information and practice in measure- 
ment statistics and research design; (2) some training in psycholo^ or 
educational psychology; (3) some active interest in liberal Lts or g English 
Scifenc ® education; (4) some background in computer programming, 
preferably a good working knowledge of FORTRAN; and (5) high scores on tests 
of graduate aptitude, such as the GRE or MAT. 



a , Clearly, it is difficult to evaluate all of the aforementioned criteria 
and certainly, one applicants conception of a "good background" might , 

drastically differ from another's. However, the staff of^preseSion felt 
unanimously that. almost all of the participants of the presession had the neces- 
sary background in order to benefit from the materials which were presented. 

What follows is a list of the participants of the presession: 



J. William Asher 
Purdue Educ. Res. Center, 
S. C. C. Building G 
Purdue University 
Lafayette, Indiana 47907 

Hubert Austin 
Office of Research 
Ball State University 
Munc ie , Ind iana 



Richard D. Bloom 
203 Overbrook Road 
Piscataway, New Jersey 

| 

E. Gil Boyer 

Research for Better Schools 
121 S. Broad Street 
Philadelphia, Penna. 19107 




Toronto 5, Ontario, Canada 
G. Phillip Cartwright 

Computer-Assisted Instruction Laboratory, 
Pennsylvania State University 
University Park, Penna. 16802 







Manuel Cynamon 
Brooklyn College 
Brooklyn, New York 

Edward R. Fagan 
Penn State University 
178 Chambers Building 
University Park, Penna. 16802 

Charles F. Harrington 

Office of Student Affairs Research 

McGuffey Hall 

Ohio University 

Athens, Ohio 45701 

James D. Hennes 
Program Evaluation Center 
406 Turner Ave., 

Columbia, Missouri 65201 

William E Hunt 

Florida Atlantic University 

Boca Raton, Florida 

Connie G. W. Meredith 
Hawaii Curriculum Center 
1625 Wist Place, 

Honolulu, Hawaii 96821 

Stan Middlestad 
Department of Design, 

University of Waterloo 
Waterloo, Ontario, Canada 

Edward W. Pepyne 
Montague House, 

School of Education 
University of Massachusetts 
Amherst, Massachusetts 
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Robert R. Prochnow 
Inter-American Educational Center, 

2525 Tower Life Building 
San Antonio, Texas 78205 

Samuel C. Reed 
California Test Bureau 
Bel Monte Research Park, 

Monterey, California 93940 

Jose R. Rios-Garau 

Centro de Investigaciones y Dat os Educativos, 

Edificio Barreras 

Calle Guayama esq. Barbosa, 

Hato Rey, Puerto Rico 

Mitchell Schorow 
Ins true. Systems Sec., 

Center for Study of Medical Educ., 

University of Illinois 

Medical Center 

901 S. Wolcott Street, 

Chicago, Illinois 60612 

Anita Simon 

Research for Better Schools, 

121 S. Broad Street, 

20th Floor 

Philadelphia, Penna. 19107 

Henry B. Slotnick 
284 Education 
University of Illinois 
Champaign, Illinois 61820 

Kenneth J. Travers 
1805 Lynwood Drive 
Champaign, Illinois 61820 

Merlin W. Wahls trom 

Ontario Institute for Studies in Education 
102 Bloor Street, West 
Toronto 5, Ontario, Canada 

Wendell W. Weaver 
College of Education 
University of Georgia 
Athens, Georgia 30601 

E. Belvin Williams 
Computer Center 
Teachers College 
Columbia University 
New York, New York 10027 
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Peter F. Wolmut 

Multnomah County Intermediate Education District 
Portland, Oregon 

Jules M. Zimmer 
Montague House 
School of Education 
University of Massachusetts 
Amherst, Massachusetts 01002 



•I 



wmmmmmmmrn 
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The following tables summarize some information about the 27 partici- 
pants of the presession on seven variables. This information was collected 
prior to the presession on the standard AERA Presession Application Form. 



Table 1 



Age of Participants 



AGE FREQUENCY 



<25 

25-29 

30-34 

35-39 

40-44 

45-49 

50-54 

>54 



0 

5 

6 
7 
7 
1 
1 
0 



Table 2 

Sex of Participants 

SEX FREQUENCY 

Males 25 

Females 2 





►i 



! ERiC 
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Table 3 

Major Field of Ph.D. Work 

Field Frequency 

1 
2 
8 

/ 1 
1 

?• 1 
2 
2 
1 
1 
1 

Not Reported or 

No Ph.D. 6 



Communications 
Ed. Administration 
Ed. Psychology 
Ed. Research 
English Education 
Math. Education 
Measurement 
Psychology 
Science Education 
Special Education 
Student Personnel 




r 



Table 4 

Percentages of Time Spent by Participants on Research and Teaching 

< 

, Frequency 

Percentage ' Research Teaching 



0-20 3 12 

21-40 7 4 

41-60 6 3 

61—80 3 4 

81-100 6 1 

Unknown 2 3 





} 
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Table 5 

♦ • 

Memberships in Professional Organizations of Partici^ mts 

(Other than AERA) 
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.A 



Xhe irre'-r loua 



Organization 

APA 

NOME 

Psychometric Society 
AEDS , ASA, PDK 
ACM, NCTM, AMA, AAAS , NEA 
AASA, NSTA, AAHP, DPMA, SPA, 

- ■ v }S&Pk, IRA# CEC, NSSE, AAMC 



Frequency 

9 . 

■"'5 

4 

3 

2 



c -I pants. 



-v, ,-vi 



u* : 



v 



/*»■*» ; '*> *V t 

W» O 1 

n: f vn ; .-f-r ;»•>' 



IJ'IS CCT 






1 

JL *;‘0 5* 



C '.‘-‘v Table 6 

Number of Funded Research Projects Directed 
or Co-Directed by Participants 



• \ '"(A :■ 






Number 

0 

1 

2 

3 

4 

5 

6 

7 

8 
9 



Frequency 

10 

11 

3 

2 

' ■ 0 

0 

0 

0 

0 

1 



e 'i 'i, 

✓:> • 






U '••• • \ 



Table 7 

Approximate Numbers of Courses Taken by Participants in Relevant Areas 

■ . 1 .w 

* 

No, of Subject Areas 




Courses 


An thro. 


Curr , 


Adm. 


Meas. 


Computers 


Linguistics Math, 


Psy. 


Socio. 


Stat 


0-1 


19 l 


10 


16 


5 


12 


24. 


4 


1 


11 • 


: i.- 


2-3 


7 


12 


? 


10 


12 


3 


3 


3 


6 


• 13. .v 


4-5 




3 


2 


8 


3 




5 


6 


• 5 ■ 


8-* 


6-7 




1 


1 


3 






4 




3 •; 


• 2 


8-9 








1 






2 


• 4 




’ • 2 


More 


- 






» 






* 


, 


' . . 


. ' « 


than 10 




1 


2 


• 


■ • 


♦ 


9 


13 


2 


1 • 






o 
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EVALUATION OF THE PRESESSIQN 



As one can readily see from the list of materials whiuh speared in 
the previous section, many evaluative devices were distributed to the parti- 
cipants. These devices were designed to serve two relatively distinct 
purposes. First, the many content mastery tests, designed, administered and 
scored by the various staff members, served to provide feedback to the staff 
members about the quality of their instruction. These instruments were 
generally administered immediately following each lecture. In many cases 
the correct responses to each item were also distributed, so as to provide 
feedback to the participants. 

Another class of instruments was designed to evaluate the presession 
as a whole. Some of these devices were administered on the first day of the 
presession as "pre-test" instruments. Others, including open-ended question- 
naires, were administered and scored after the second complete day of the 
presessions. This was done so that the presession organization and/or proce- 
dures could be changed if needed. Finally, instruments were administered at 
the close of the presession. These were post-tests, corresponding, in some 
cases, to the pre-tests given earlier. 

Of all these instruments, five are presented and discussed in this 
section. 

The first, labeled "Instrument I-C" asked the participants to rate 
each of 30 terms and concepts associated with natural language computing on 
each of five scales. The scales, dealing with the usefulness, complexity, 
practicality, and popularity of the concept, yield a fair estimate of the 
subjects conception of the concept. This instrument was administered at the 
beginning and at the end of the presession so that changes in the partici- 
pants could be determined. The results, reported on the scales, indicate that 
the participants perception of these concepts had almost universally improved 
during the course of the five-day meeting. 

The second instrument, labeled "Instrument II-B" was also administered 
twice during the presession. The first administration, after the second day 
of the meeting, provided information about the general satisfaction of the 
participants with the way in which the presession was conducted. The second 
administration, at the end of the presession, yielded similar information 
for the entire meeting. As can be seen by the results reported on the instru- 
ments, the participants were quite happy with the way the meeting was conducted. 
The results of the second administration indicate that this general satisfaction 
increased throughout the meeting. 



The third instrument, labeled "Instrument I-B" asked the partici- 
pants to rate their abilities to perform each of 30 tasks; all related to 
natural language computing. This instrument was also administered twice, 
once at the beginning and once at the end of the presession. The differences 
in the mean ratings given each item indicate that in general the participants 
perceived an increase in their abilities to perform these various tasks. 

There are two exceptions to this. First, if most participants felt that they 
could perform a task quite easily at the onset of the presession, there was, 
of course, no change in responses. Second, the items on which subjects 
showed little change are those that contain content that was not explicitly 
discussed during the presession. 

The fourth and fifth instruments, named "Participant Evaluation Form": 
and "Presession Critique for Staff Members" are standard AERA. instruments 
and were administered only once, at the close of the presession. The tab- 
ulation of responses oi the fourth instrument, again indicates a general 
satisfaction with the presession. The few responses indicating dissatis- 
faction generally refer to one or two specific sessions which could have been 
improved. The fifth instrument, the staff evaluation form, shows similar 
results . 

In summary, the results obtained by all of these instruments indicate 
a general professional growth on the part of both the participants and the 
staff, as well as a general satisfaction with the presession. 



Instrument I-C 



Below you will find a series of words and phrases, each representing 
a concept, procedure, or language. Immediately below are five scales. For 
each word please write that value for each scale which most closely reflects 
your feeling about the concept, procedure, or language. Write these values 
on the lines immediately following each word so that the first value indicates 
your choice on the first scale, the second value your choice on the second 



scale, etc. 

Scale 1 


1 




2 


3 




4 


5 








Useful 










Useless 






Scale 2 


; 






3. 


• 




5 








Simple 










Complicated 






Scale 3 


1 




2 


3 




4 


5 








Practical 








Theoretical 






Scale 4 


1 




2 


3 




4 


5 








Well known 








Little known 






Scale 5 


; 




.? 


3. 




4 


5 








Interesting 






1 




Dull 




1. 


Binary search 






6. 


Pattern recognition 




(S) 


1.8 2.1 


1.8 


3.0 


2.9 


(4) 


1.6 


3.6 2.6 


3-3 


1.8 


(1) 


1.1 2.0 


1.2 


2.0 


2.0 


(0) 


1.6 


4-1 2.5 


3.3 


1.6 


2. 


PI/I 








7. 


Alphameric arrays 






(3) 


hA hi 


1.9 


hi 


2.0 


(?) 


1.3 


1.8 1.4 


2.6 


2.8 


(1) 


1.1 2.9 


1.2 


2.8 


1.3 


(1) 


1.0 


1.6 1.1 


1.7 


2.7 


3. 


Symbolic logic 






8. 


List processing 






(2) 


1.8 3.5 


hi 


hi 


hi 


(S) 


hi 


2.8 1.8 


hi 


2.5 


(1) 


1.6 3-5 


2.3 


3.2 


2.0 


(0) 


1.1 


3.0 1.7 


2.6 


1.6 


4. 


Computer parsing 






9. 


Predicate calculus 






(12) 


1 hi hi 


3.2 


4.2 


2.6 


(IS) 


3.3 


4 • 0 4*5 


4.5 


2.6 


(2) 


1.7 3.8 


2.3 


3.3 


1.9 


(1) 


1.6 


JTs 2 T 4 


3,6 


2.1 


5. 


Content analysis 






10. 


Information retrieval 




(3) 


1*2 2*2 


1.8 


2.8 


hi 


(2) 


1.0 


3.7 1.6 


2.3 


1.8 


(1) 


1.2 3.4 


1.7 


2.4 


1.3 


(1) 


1.1 


3.6 1.5 


2.4 


1.6 




HB mm 



mmmmm 
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11. 


Artificial 


intelligence 




(2) 


1.9 


4-4 


3.5 


3.8 


1.4 


(1) 


2.2 


4.3 


3.5 


3.5 


1.5 


12. 


LISP 










(14) 


2*2 


2*2 


2.2 


L -2 


3. A 


(2) 


1.7 


3.2 


1.9 


3.1 


2.1 


13. 


Automata theory 






(17) 


2.0 


4.1 


4-2 


4.2 


1.9 


(2) 


2.2 


3.3 


3.4 


3.8 


1.7 


14. 


Recursive functions 




(14) 


1.9 


2.9 


2.4 


3.5 


2.5 


(3) 


1.6 


3.1 


2.3 


3.3 


2.0 


15. 


Dynamic storage 






(14) 


1.3 


3.5 


1.7 


3.8 


2.2 


(3) 


1.2 


2.9 


1.3 


3.2 


2.0 


16. 


Nash coding 








(22) 


3.7 


3.7 


3.3 


4* 6 


3.3 


(3) 


1.8 


2.8 


2.0 


3.2 


2.1 


17. 


Contingency analysis 




(18) 


1. 6 


3.0 


2.6 


3.5 


2.7 


(7) 


1.6 


2.8 


1.9 


3.3 


2.2 


18. 


Algeriths 








(4) 


1.2 


2.1 


1.8 


1.9 


1.9 


(2) 


1.0 


2.3 


1.3 


1.7 


1.2 


19. 


Truth-table 








(12) 


1.8 


2.1 


2.3 


2.7 


2.1 


(2) 


1.6 


2.7 


2.4 


2.9 


1.7 


20. 


Turing machine 






(14) 


2.6 


2.8 


4.1 


3.1 


1.9 


(2) 


2.3 


2.0 


3.1 


3.0 


1.5 



21. 


General 


Inquirer 






(10) 


1.4 


3.3 


1.9 


2.7 


1.7 


(2) 


1.5 


3.1 


1.7 


2.4 


1.4 


22. 


PEG 










(IV) 


1.6 


3.7 


2.2 


3.3 


1.6 


(3) 


1.5 


3.0 


1.8 


2.8 


1.2 


23- 


Heuristics 








(ID 


1.1 


3.6 


2.0 


3.0 


1.6 


(5) 


1.2 


3.2 


2.1 


3.3 


1.3 


24- 


Polish notation 






(22) 


3.0 


2.5 


3.2 


4-5 


3.5 


(6) 


1.9 


2.1 


2.3 


3.5 


2.0 


25. 


Context- 


-free 


grammar 




(19) 


1.7 


4.0 


2.0 


4.0 

WMWI 


1.8 


(3) 


2.1 


2.9 


2.5 


3.3 


1.5 


26. 


PR0TRAN 










(22) 




2*2 




Ail 


1 2*2 


(5) 


2.3 


3.0 


2.2 


3.8 


**”"2.0 


27. 


Push-down 


store 






(19) 


2.2 


3.5 


2.3 


4.1 


2.8 


(6) 


1.8 


2.4 


2.0 


3.0 


2.4 


28. 


MAPTEXT 








(22) 


2.0 


4.0 


2.7 


4-7 


3.3 


(12) 


1.9 


2.6 


2.4 


4-1 


2.3 


29. 


SC0RTEXT 








(22) 


3.0 


4-3 


3.3 


4.7 


3.7 


(6) 


1.5 


2.9 


1.8 


4.0 


1.7 


30. 


SN0B0L 










(14) 


1.5 


3.3 


2.3 


4.0 


3.3 


(5) 


2.1 


2.8 


2.0 


3.5 


2.5 



Note: Numbers above each line are pre-test scores. Numbers below are corres- 

ponding post-test scores. Numbers in () indicate number of participants who 
did not respond to the item. 
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Instrument I-D 



HOW WELL COULD I DO 



This instrument is designed to determine how well you could 
perform a series of tasks, according to your perceptions of your 
abilities. Would you please read each of the following tasks and 
rate your abilities to handle them on a i-5 scale as follows: 



i r 



(1) 

( 2 ) 

(3) 

(4) 

(5) 



I could easily do this. 

I would have some difficulty in doing this. 

I would have a great deal of difficulty doing 
this, but could probably get it done. 

I could probably not do .this. 

It would be hopeless for me even to attempt to 
do this task. j 



(PLEASE MARK ALL YOUR ANSWERS ON THE SEPARATE ANSWER SHEET.) 



1.8 

1.8 

2.0 

1.8 

2.2 

2.0 

2.8 

2.1 

4.0 

2.9.9 

3.3 

2.0 

3.5 
2.1 

2.8 

2.1 

2.6 
2.2 

2.9 

1.4 

,V9 

5 

4.1 

2.3 



1. Write a FORTRAN II program that calculates means and . 
standard deviations. 

* 

2. Write a FORTRAN IV program that does the above. 

3. Write a PL/I program that does the abpve. 

4. Write a FORTRAN IV program that determines whether or not 
a number is contained in a given list of numbers. 

5. Write a FORTRAN IV program that does (4) for natural language 
words rather than for numbers . 

6. Write a PL/I program that will do (5). 

j * . * ■ 

7. Obtain a relatively complete list of biographical items that 
deal with computer programs that play chess. 

8. Write a FORTRAN II or. IV program that performs a binary search..' 

» • 

9. Write a FORTRAN II or IV program that places a vector of 
numbers in numerical sequence. 

10. Write a FORTRAN II or IV program that arranges a list of words ! 
in alphabetical order. 

11. Identify 5 professional organizations whose journals arc 

relevant to natural language computing. v 

12. Write a brief paper which discusses recent progress (say 
within the last 5 years) in computer parsing. 

i f T , 

13. Complete a truth-tabic for a fairly complex statement in the 
propositional calculus. 



o 

ERIC 



• i 



9MPV 
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is 

3.7 
1.9 
4.5 

3.0 

1.8 
1.2 

3.1 

1.7 

2.4 

1.5 

3.2 

2.8 

2.3 
2.2 

4.1 

3.5 

3.5 

3.2 

3.2 

2.3 



3.8 

2.4 



4.2 
2.5 

3.4 

2.5 

3.7 

2.8 

3.7 

2.7 

4.3 

3.4 



14. Discuss, in some detail, what some of the major problems 
in pattern recognition are. 

15. State under what conditions the, statement -(pv(qAr)) is true. 

16. Explain how SN0B0L differs from LISP. 

17. Explain what an algorithm is. 

18. Show under what types of circumstances a list processing 
language is to be greatly preferred over other languages. 

19. Locate a running cononical correlation program. 

20. Compare the relative speed of PL/I and FORTRAN IV programs. 

• • ✓ * 

21. Explain the difference between probabilistic and deterministic 
strategies in computing. 

22. Show why a question can not be considered a statement in the 
prepositional calculus. 

23. Evaluate the relative merits of various time sharing systems. 

24. * Write the necessary FORTRAN IV statement(s) to branch to 

statement 100 when the following expression is true 
-(pv(qAr) ) . , 

25. Discuss the efficiencies of various list look-up strategies . 



Name at least three leading researchers working in the following 
areas : 

i ’ ’ , . 

26. Stylistic analysis 

27. Content analysis 

28. Information retrieval 

29. Verbal lec?.rning and the computer 

30. Theory of algorithms 



Note: The first value preceding each item is the pre-test mean. 

The second value is the post-test mean. 
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Instrument II-B 



O 



Key: SA (Strongly Agree) r A (Agree) , ? (Undecided) , D (Disagree) , 

SD (Strongly Disagree) , NA (No Answer) . Please circle your choices. 



1 . 

2 . 

3 . 

4 . 



The objectives of this program 
were clear to me 



The objectives of this program 
were not realistic 



The participants accepted the 
purposes of this program 



5 . 
6 • 



The objectives of this program 
were not the same as my 
objectives 

I have not learned much new 



The material presented seemed 
valuable to me 



7 . 

8 . 

9 . 

10 . 

11 . 

12 . 

13 . 

14 . 

15 . 

16 . 



I could have learned as much by 
reading a book 



Possible solutions to my problems 
are not being considered 



The information presented was too 
elementary 

The speakers really knew their 
subjects 

I was stimulated to think about 
the topics presented 

We worked together well as a 
group 



The group discussions were 
excellent 



There was little time for 
informal conversation 



17 . 

18 ,. 



1 9 • 

( ) ' 



I had no opportunity to express 
my ideas 

I really felt a part of this 
group 

My time was well spent 

The program met my expectations 

Too much time was devoted to 
trivial matters 



20 . 



The information presented was 
too advanced 



3 

ERIC 



SA A ? D SD NA SA A ? D SD NA 



10 8 13 2 



2 4 12 6 



5 13 6 



1 3 5 8 7 

2 7 15 



13 10 1 



2 2 10 10 



1 3 6 8 4 2 



1 1 11 11 



18 6 
14 9 



4 9 8 2 1 



1 4 14 4 



1 3 2 14 3 



2 2 14 5 



3 15 3 

10 12 2 
10 10 2 



4 3 14 3 



1 

1 

1 

1 



1 2 4 10 6 1 



12 10 1 1 



19 5 



20 4 



1 1 



4 10 10 



8 15 1 



3 4 7 10 

1 3 20 



19 4 1 



9 14 



1 3 13 61 



1 10 13 



8 13 2 



8 7 7 1 



2 2 14 6 



1 10 13 



6 17 1 

17 7 

10 11 2 1 



1 1 14 8 



17 5 
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U 



SA A ? D SD NA SA A ? D SD NA 



21. The content was not readily appli- 
cable to much research in education 

22. The Assistant was very helpful 

23. Theory was not related to practice 

24. The schedule should have been more 
flexible 



2 4 5 13 

5 6 12 1 

3 4 5 7 3 2 

2 . 4 11 6 1 



u 
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2 10 12 
11 8 3 11 

2 2 9 10 1 

4 3 10 7 
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PRESESSXQN CRITIQUE FOR STAFF MEMBERS 



>• 

u 

O 



Indicate your observation and judgment by checking each item In one 
column at the left. Items not applicable or not subject to your ob- 
servation should be omitted. Be frank. 



I. Environmental conditions 

a. Classroom spaces •: ? 

Work spaces • . i; . . 

Living quarters 

Teaching equipment, aids (chalk boards, pub I ic address system, etc.) 
Resource material, library | 

Eating facilities ... . j\ 



5. 



b. 

c. 

d. 

6i 

f. 



Participants 

a. Appropriateness of academic backgrounds 
Sufficiency of research experience 
Will ingness to work 
Intellectual curiosity : - 
Concern for app I i cab 1 1 i ty of techn i ques 
Aspiration 

Immediate preparation for Presession 



b. 

c. 

d. 

e. 

f. 
9* 



* V • • 



Organization 

a. Adequacy of notice to prospective app! i cants 
Sufficiency of preplanning 
Smoothness of operation 
Adaptability to obstacles and feedback 
SensL’vlty to grievances 
Adequacy of financial support 



b. 

c. 

d. 

e. 

f. 



4. Schedule 



a. 

b. 

c. 

d. 

e. 

f. 

9* 

h. 

i * 

j* 



Appropriateness of five days for the job 
Time spent efficiently 
Events sequenced appropriately 
Punctua I 1 ty 

Balance between formal, informal affairs 
Quantity of discussions 
Quality of discussions 
Quality of formal presentations 
Unobtrusiveness of evaluation efforts 
Methods of evaluation 



Outcomes 

a. Intended content was actually taught 
Increase in participant understanding 
Improvement in attitude toward research 
Personal associations initiated 



(Over) 
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6. In general was the Presession well organized? 



(5) Yes 



(1) No Answer 



7. Were the facilities suitable for the activities which you had planned? If not, 
specify. 



8. Should Presessions be limited to the same hotel, or the same city, in which the 
annual meetings will be held? 



9. Were you to do the same assignment over, in what major ways, if any, would you 

change your contribution? A 

(1) Greater depth (2) No (1) Adhere more closely (2) No Answer 

to schedule , , 

10. Do you wish that the D.i rector had made firmer arrangements to assure partici- 
pants and you of the staff opportunity to meet in pairs or small groups? < 

(4) No (2) No answer 

11. Were the objectives you set for yourself during the Presession attained? .. 

(5) Yes (1) No answer 

12. Are you inclined to urge your colleagues to become staff members for. such an 
institute or Presession? 

(5) Yes (1) No answer 

13. In what ways, if any, did you as a staff member benefit personally as a result 
of your participation in this Presession? 

1. Meeting new people who are interested in the area. * • 

2. Own growth in professional competence. : ,f 

3. Learned from other staff members. 



(5) Yes 



(1) No Answer (2) Too Expensive 



(2) Yes 



(1) No (1) Probably Yes (2) No Answer 
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PARTICIPANT EVALUATION FORM 

) 



Pi recti ons : Please respond with a word, a phrase, or one or more sentences to as 

many of the following questions as you can. Your frank and honest evaluation can 
only benefit everyone concerned. Po not identify yourself by name unless you prefer 
to do so. 

Environment and-Faci 1 i ties 

1. a. To what extent : d i d the relative availability or unavailability of books and 

journals interfere with or promote your attempts to master the content of 
this session? 

90% - not at all 5% - a little- 5% - the library helped greatly 

b. To what extent did reproduced materials given to you by the staff improve matters! 
90% - a lot; greatly; etc. 10%. - helped greatly,, but insufficient 

’ time to study., these materials 

2. a. Did you feel that you lacked a "place to work," either alone or in small groups? 



97% - No .. 3% - "sort of half" 

b. Was your room satisfactory? 

94% - Yes , 6% - np ventilation, too expensive ... 

3.' a. Which features of the meeting rooms were inadequate or not conducive to learning? 
90% - none 10% - seats, noise, blackboards, coffee 



4. 



5 . 




. 7, 




6 . 



b. Which features were especially facilitative in the same regard? 

carpeting, tables, water, quiet, the "happy hour" suite, comfort. 

Scheduling and Organization . nothing 

a. Was five days too long a period to leave your work at home for the purpose of 
attending this session? 

94% - no ' 6% - yes , 

b. Was five days too short a period in which to learn much of the content of 
this session? 

.50% - no -50%. -. yes • 

a. Were you at lowed enough time in which to pursue activities of your own choosing? 
85% - yes. 15% - no 

b. Would you have preferred hot to meet in the evening after dinner? 

94% - no 6% - yes 

c. Would you have preferred more or fewer meetings per day than there actually 
were? Or was the number. of meetings per day agreeable to you? < 

94% - agreeable 6% - a little tight 

a* Were the. individual lectures too long to. sit and listen or take notes? 

60% - no . 40% - some ; 

b. Were the lectures scheduled in an appropriate sequence? 

, 97% - yes . . 3% - "didn't 

Did you have sufficient opportunities to 

97% - yes 3%,.- barely 

a. Were the instructors too inaccessible or unapproachable so that you did 
not get the individual attention that you desired? 

100% - no 



notice" 

interact with other participants? 



b. Was it helpful to have graduate student assistants present? 
97% - yes 3% - "undecided" 



o 



ERIC 



9. Did the attempts to evaluate your progress and reactions during the session 
(and at this moment) interfere with your work here? 

90%-no 3%-a little 3% - found them irritating 3%. found 

them 

10. In general, was the Presession well organized? distract- 

100% - yes ing 

Content and Presentation 

11. a. Did the content of the lectures and readings presuppose far more previous •• 

training than you had? ’•* . 

. i • • 

70% - no 20% - occasionally 10% - yes 
b. Should less training in these areas or more have been presupposed? ■ * 

70% - neither 20% 7 more training 10% - can't say 

12. To what extent was the content of the lectures and readings relevant to what 

you hoped to accomplish during the session? 

75% - greatly 15% adequately 10% - no answer 

13. a. Were the lecturers stimulating and interesting? 

Yes, or generally - 100% 

b. Were the lecturers competent to speak on the subject assigned them? 

Yes, or generally - 100% 

* . / 

c. Were the lecturers we I I prepared? 

Yes, or generally - 100% 

# # 

14. Were you disappointed in any way with the group of participants? 

97% - no 3%-a little ’ 

Answer each of the following only by checking the more appropriate blank: 

» 

15. If you had it to do over again would you apply for this Presession which you 1 

have just completed? Yes 100% No 

16. If a presession such as this is held again would you recommend to others like 
you that they attand? Yes 100% No 

v 

17. Do you anticipate maintaining some sort of contact with at least one of the 
Presession staff? Ye s 94% N o 6 % 

18. Do you feel that AERA is making an important contribution to education by 
sponsoring presessions such as this one? Yes 100% No (many indicated strong 

c^.q]T 6 cment ) 

19. Do you feel that anything has happened during these five days to make iT more 

likely that you will leave your present position of employment? Yes N o94% - 6%N.i 

20. Is it likely that you will collaborate in research with someone else attending.' 
this Presession (other than those you already were likely to collaborate with)? 

Yes 40% No 60% 

21. Do you feel that the staff should feel that it has accomplished its objectives 

during this five-day presession? Yes 97% No 



3% 



"what were the objectives??" 



GENERAL COMMENTS BY THE DIRECTOR OF PRESESSION 



What struck me from the beginning was the irregular and voluntaristic 
nature of our presession. The others planned by AERA this year, and supported 
by the USOE, seemed to have an identifiable professional clientele ready-made 
for them, with extensive professional training already gained. This is true of 
the various statistical groups, ones concerned with college student personnel 
and similar substantive fields. And the payoff for such work is already well 
established: The statistical rules are the rules of grantsmanship ; experi- 

mental design is in constant demand; personnel research positions abound, etc. 

In contrast, our presession dealt with an esoteric, almost non-esistent 
interface of disciplines, and with a kind of research which (at least in 
education) must have a glorious future, but almost no present or past at all. 
Nevertheless, ‘there is a way to present the work publicly so that workers will 
identify with it. But through a mixup (and my being in Europe during a 
critical time), the regular announcement did not make this kind of identific- 
ation. Rather, there was a quite forbidding announcement, mentioning "computa- 
tional linguistics," "parsing," and various other recondite terms which are 
almost guaranteed to frighten off all but a particular, unafraid sort of 
participant. 

Thus we found our participants to be an unusual group of people: 
unusually qualified in mathematics; not afraid of formalisms and abstractions; 
generally eager to learn new things they have hardly even heard of before; 
tolerant of occasional obscurity, and of our inevitiable difficulties in 
teaching certain things which we may never have studied formally ourselves, 
and which we might be teaching for the first time. 

^ The participants were also convivial, and the majority of them (though 
not all) early established and maintained, both with the staff and with each 
other, an easy first-name friendliness which, hopefully, helped to create 
something of a subculture in natural language processing for education. This 
belief is supported by their ratings and by the fine letters we have received 
since the end of the presession. We enjoyed it, and learned to like and respect 
each other, along with the emerging discipline which brought us together. While 
the days were often spent in learning technique, the evenings (after the 
sessions in enrichment or background areas) were often spent in spirited dis- 
cussion of relevant philosophy (the limits of simulation, the mind/machine 
paradox, the programming of semantic problems, etc.), which are probably 
essential to a new discipline, too. The group even discovered a very talented 
guitarist and folk-singer in their midst! 

From. my own biased viewpoint, I would agree with the apparent consensus 
of the participants that the presession, though highly experimental, was a 
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distinct success, and that the topic deserves a recurring place in the 
future plans of the AERA. It is surely one of the freshest research fields, 
with both methodological and substantive importance, and it promises to 
retain some of its experimental glow in the rapidly changing computer world 
of the next few years, even though workers and projects are certain to 
multiply in the profession. 

For the future, we can count on fast changes in relative importance 
of various topics. The leaders, in another year or two, could probably 
ass um e a participant familiarity with PL/l, which this year was little 
known, and such a headstart would permit more rapid development of actual 
algorithms. And the field of artificial intelligence will probably be better 
articulated, in its relation to educational research. 

Probably more of the potential participants in educational research 
will already have close ties with the main stream of computer science (as 
represented in ACM, ASIS, etc.) than is now the case, and this movement would 
suggest a shift in part of the curriculum. And time-sharing terminals will 
be richer in software convenience than was our QUIKTRAN this year, so that 
natural-language macro-instructions may be given more on-line demonstration 
and practice. 

I would like to express my profound thanks to the staff and partici- 
pants who made this possible, and to AERA personnel, both central office 
and membership, who helped us so much, and to the IBM Corporation for the 
free terminal use. And to Dieter Paulus, for essential help all the way, 
through the editing of this report. Knowing what we know from this first 
experience, we could surely improve the program in some solid ways in another 
year. But I can't help wondering whether it could again be as enjoyable. 

— E. B.. P . 
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INSTRUCTIONAL PRODUCT DEVELOPMENT 



Directors 

Dr. W. James Popham, University of California - Los Angeles 
Dr. Howard J. Sullivan, Southwest Regional Laboratory for Educational 

Research & Development 



INTRODUCTION AND OBJECTIVES 



The past two yearr have brought a sharp upsurge of educator interest 
in the procedures associated with the systematic development of educational 
products that achieve pre-specif ied instructional objectives. A concomitant 
of the recent support for the development and evaluation of new instructional • 
programs has been an increased demand for skilled product developers. Yet, 
due in part to the recency of the product development movement, individuals 
with special competence in this area are in short supply. 

The overall goal of the 1968 AERA Presession, Instructional Product 
Research , was to develop the participants’ skills in product development and 
research. The procedures used in planning, conducting and evaluating the 
presession parallel those employed in programmatic product development efforts. 
Initially, the specific competencies required for the conduct of successful 
product development and research operations were carefully analyzed by the 
presession staff. The instructional objectives and evaluation procedures for 
the presession were derived from this analysis. Instructional materials and 
activities designed to implement the pre-specif ied o' ‘actives were then 
either prepared by the presession staff or selected from among product- 
developer training materials produced at UC.'A or at the Southwest Regional 
Laboratory for Educational Research and Development. 

The analysis of important product development skills yielded a list of 
12 instructional objectives. Participants who mastered these objectives were 
able to perform the following tasks at the conclusion of the five-day presession 



1. Identify and specify the objectives for instructional products in terms 
of observable learner behavior. 

2. Classify instructional objectives according to a modified, four-category 
version of the Taxonomies of Educational Objectives . 

3. Discriminate among instructional objectives which contain (a) no minimal 
levels of learner behavior, (b) minimal levels of behavior only for an 
individual learner, and (c) minimal levels of behavior for a group of 
learners . 



4. Select and/or write suitable test items, given precise instructional 
objectives . 



\ • • •* 

5. Correctly classify (according to a four-category scheme) different types 
of criterion measures which may be used to evaluate educational products. 
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6. Correctly classify different components of sample instructional specifi- 
cations. 

7. Sequence enroute behaviors according to prescribed criteria. 

8. Correctly identify instances in which product development rules have 
been (a) followed or (b) violated when presented with a series of 
fictitious vignettes describing product development activities. 

9. Identify in given instructional products examples of selected develop- 
ment principles. 

10. List and briefly describe appropriate procedures in product tryout and 
revision. 

11. Specify criteria useful in evaluating the effectiveness of product 
development operations. 

12. Recommend appropriate research designs for hypothetical situations 
requiring the experimental evaluation of instruct lonal products. 
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LIST OF STAFF 



Co-Directors: 

W. James Popham 

Education Department 

University of Calif ornia-Los Angeles 

Howard J. Sullivan 
Southwest Regional Laboratory for 
Educational Research and Development 



Instructors : 

Eva L. Baker 

Southwest Regional Laboratory for 
Educational Research and Development 

Richard E. Schutz 
Southwest Regional Laboratory for 
Educational Research and Development 



Assistant: 

Leslie Bronstein 

University of Cal if ornia-Los Angeles 



I PARTICIPANTS 



An approximate total of 75 completed applications were received for 
the Instructional Product Research Presession. Letters of acceptance were 
mailed to the first 56 applicants. All subsequent applications were rejected 
because the only available meeting room at the presession he tel could reason- 
ably accommodate a maximum of 50 people. Withdrawal by some accepted appli- 
cants prior to the presession and the failure of others to appear at the 
meetings reduced the number of actual participants to 47. A list showing the 
name, address, institutional affiliation and job description for each parti- 
cipant is presented on the following pages. 

To supplement the list of participants, additional descriptive data 
were tabulated from the applications. These data are presented in the 8 
numbered topics immediately below. 

1. Average age of the 47 participants: 41 

2. Sex: 35 males, 12 females 

3. Number holding doctorate: 36 

4* Number who have had one or more funded projects: 22 

5. Average number of funded projects for all participants: 1.1 

6. Number who have had one or more publications (as determined by 

item 14 on application form) : 37 

7. Average number of publications for all participants: 8.3 

8. Institutional affiliation j 

a. Colleges and Universities: 31 

b. Regional Laboratories: 6 

c. Public Schools: u 

d. Business and Industry: 3 

e. National Educational Research Bureau 

i (Director of Educational Research, Sweden): 1 



LIST OF PARTICIPANTS 
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Title or 


Name 


Address 


Affiliation 


Position 


f 






Description 


v Allen, William H. 


Dept, of Instructional Tachnolgy 


University of 


Professor of 




School of Education 


Southern 


Education 


i 


University of South. California 


California 






Los Angeles, 90007 






Berman, Marlene 


3750 Woodward Avenue 


Michigan -Ohio 


Research & 




Detroit, Michigan 48235 


Regional Edu- 


Teaching 






cational Lab. 




Bernazza, Ann Marie 


U-93 University of Connecticut 


University of 


Research 




Department of Ed. Psychology 


Connecticut 


Associate 




Storrs, Connecticut 


* 




Bingman, Richard M. 


10601 Ease 65th Street 


Mid-continent 


Educational 




Raytown, Missouri 64133 


Regional 


Program 






Educational 


Specialist 






Laboratory 




Blaney, Jack P. 


Extension Department 


University of 


Assoc. Dir. c 




University of British Columbia 


British 


University 




Vancouver 8, B.C. Canada 


Columbia 


Extension 



Broadbcnt , Frank W. College of Education 

Drake University 
De sMo ine s , Iowa 



Drake University Assoc. Prof esse 

of Education 



Carter, Heather L. 


7401 New Hampshire Avenue, #907 
Hyattsville, Maryland 20783 


University of 
Maryland 


Research and 
Teaching 


Carr, Julian W. 


800 Washington Avenue 
Minneapolis, Minnesota 


Readers Digest 

Educational 

Division 




Champoux, Ellen M. 


School of Home Economics 
University of North Carolina 
Greensboro, North Carolina 27412 


University of 
North Carolina 


Graduate 
Teaching & 
Advising 


Dixon, James E. 


Physics Department 
Iowa State University 
Ames, Iowa 50010 


Iowa State 
University 


Instructor in 
Physics 


^ixon, W. Robert 


! 

School of Education 
University of Michigan 
Ann Arbor, Michigan 48104 


University of 
Michigan 


Professor of 

Educational 

Psychology 


Erickson, Richard C. 


1722 Summit Drive 

West Lafayette, Indiana 47906 


Purdue 

University 


Teacher under- 
graduate & 
graduates 
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Evans, Ross A. 

Finder, Morris 
Gezi, Kalil I. 

Groff, Warren H. 

Hainill, Charles 0. 

Hanson, James R. 
Hopson, James A. 

Kliger Samuel 

Koos, Eugenia M. 
Linden, Kathryn W. 

Lux, John E. 
McDaniel, Ernest D. 




McElhinney, James H. 
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Box 89 

Teachers College 
Columbia University 
New York, New York 10027 


Columbia 

University 


Research on 
Education of 
Handicapped 


School of Education 
SUNY 

Albany, New York 12203 


SUNY -Alb any 


Assoc. Profess< 
in English 
Education 


Chico State College 
Chico, California 95926 
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Chico State 
College 


Assoc. Profess< 
in departments 
of education & 
sociology 


i 

721 Highland Avenue 
Jenkintown, Pa. 19046 


County Office 
Doylestown, Pa. 


Administrator 
of an ESEA Tit] 
III project 


GPO Box 708 

San Juan, Puerto Rico 00936 


Puerto Rico 
Department of 
Education 


Director of j 

Office of 
Evaluation 


7824 Pearson Way N.E. 
Fridley, Minnesota 55432 


3-M Company 


Behavioral 

Scientist 



Mid-continent Regional Edu- 
cational Laboratory 
104 East Independence Avenue 
Kansas City, Missouri 64106 



Mid -continent Associate 

Regional Director 

Educational 
Laboratory 



MIND Inc. 

18 W. Putnam Avenue 
Greenwich, Connecticut 



MIND Inc. 
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Director oi 
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product design 



4907 Neosho McRel 

, Mission, Kansas 66205 



Research and 

Evaluation 

Specialist 



Educational Psychology, SCC-G 
Room 53 

Purdue University 
Lafayette, Indiana 47907 

5100 Leighton Avenue 
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Purdue Educational Research 
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Purdue University 
Lafayette, Indiana 47907 

Ball State University 
Muncie, Indiana 47306 I 



Purdue 

University 



University of 
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University 



Ball State 
University 



Teaching & 
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Teach Research 
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curriculum 
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Mehlinger, Howard D. 
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Olsen, Maurice D. 

Overing, Robert L. 
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College 



High School Curriculum Center Indiana 

in Government University 

1129 Atwater Avenue 
Bloomington, Indiana 47401 

Instructional Research Laboratory University 
202 State Street i Wisconsin 

Madison, Wisconsin 53706 
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144 Kylewood Place Ball State 

Muncie, Indiana 47304 University 



1811 Parker Street Far West Reg. 

Berkeley, California 94703 Lab. for Ed. 

Research & Dev. 



Bureau for Physically Handicapped 
Children I 

Room 870 

State Education Bldg. 

Albany, New York 12224 



N.Y. State 
Special Ed. 
Instructional 
Materials 
Center 



Faculty of Education 
University of British Columbia 
Vancouver, R.C. 



University of 

British 

Columbia 



132 W. Market Street 
Salinas, California 93901 

Science Education Center 
Sutton Hall 227b 
University of Texas 
Austin, Texas 

I 

School of Education 
State University of N.Y. 
Buffalo, New York 
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Curricula Studies 
550 City Center Bldg. j 

220 E. Huron 

Ann Arbor, Michigan 48108 

420 S. Buchanan 
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University of 
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State University 
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Director of 
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project 
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INSTRUCTIONAL PROGRAM AMD MATERIALS 



The instructional program and materials were organized to provide 
direct instruction and practice on the presession objectives. Instructional 
sessions were scheduled daily from 9:00 a.m. until 4:30 p.m. , and evening 
meetings with optional attendance were held from 7:00 until 9:00 p.m. on 
the first three nights of the presession. 

I 

The complete schedule for the presession is presented in Table 1 on 
the next page. The daytime program was divided into four sessions daily. 
Typically, direct instruction and practice were provided on the presession 
objectives during the first three sessions each day. During the final day- 
time session each participant worked individually on instructional handouts 
given that day, prepared his own small-scale instructional product, or 
consulted with presession staff members on his ongoing or planned product 
development and research activities. 

The optional evening meetings were scheduled to provide participants 
with an opportunity to interact with each other and the j re session staff in 
group discussions. Tape -films trip instructional programs were shown at the 
beginning of the first two evening meetings to stimulate the discussions. 
Attendance at the optional evening meetings ranged from a low of 25 individuals 
(53$ of the 47 presession participants) at the initial session to a high of 
40 participants (85$) at the final meeting. 

A total of 38 separate handouts, ranging in length from one to 60 pages, 
comprised tne materials used for instructional and evaluation purposes. A 
complete listing of the materials provided for all presession participants is 
presented in Table 2. 



PRESESSION SCHEDULE 
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PRESESSION MATERIALS 



1. Pre Tests: Form X, Parts I and II (20 pages & 5 pages) 

2. Pre Test Correct Answer Key (1 page) 

3. Profile Sheets (1 page) 

4. Instructional Product Research: Introductory Hand-outs (8 pages) 

5. Educational Objectives Program Answer Sheets (1 page) 

6. Objective 3 Practice Sheets (1 page) 

7. Product Documentation and Review Guidelines (10 pages) 

8. Instructional Objectives (17 pages) 

9. SWRL Technical Glossary (8 pages) 

10. Establishing Performance Standards (17 pages) 

11. Selecting Appropriate Educational Objectives (19 pages) 

12. Product Research: A New Curriculum Specialty (5 pages) 

13. Improved Educational Program Answer Sheets (1 page) 

14. Improved Educational Program Criterion Tests (1 page) 

15. Criterion Measures Examples (1 page) 

16. Educational Criterion Measures Answer Sheets (1 page) 

17. Objective 6 Practice Sheets (1 page) 

18. Rules for the Development of Instructional Products (60 pages) 

19*. Design Specifications: Objectives and Prototype Items (21 pages) 

20. Writing Instructional Specifications (15 pages) 

21. Educational Criterion Measures (24 pages) 

22. Write is Right (5 pages) 

23. Sequencing Enroute Behaviors (25 pages) 

24. Exams: Research in the Schools - Part I (4 pages) 

25. Answer Sheets: Classifying Educational Research Studies (1 page) 

26. Criterion Test Items: Objective 12 (1 page) 

27. Appropriate Practice (17 pages) 

28. Providing Knowledge of Results (17 pages) 

29. Make it Interesting (14 pages) 

30. Avoid Irrelevancies (13 pages) 

31. Exams: Research in the Schools - Part II (6 pages) 

32. Answer Sheets: Interpreting Research Results (1 page) 

33. Simplified Designs for School Research (26 pages) 

34. Developing the "D ,E in Educational Research and Development (11 pages) 

35. Research, Development, and Improvement in Education (16 pages) 

36. Post Tests: Form Z: Parts I and II (22 pages & 5 pages) 

37. Post Test Correct Answer Keys (1 page) 

38. AERA Evaluation Forms (1 page) 



'* 1 m * 
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EVALUATION RESULTS 



The primary evaluation source was the post-instructional performance 
of the participants on the instructional objectives of the pre session. Two 
parallel test forms containing 190 items each were employed as direct measures 
of the 12 objectives. Each test form measured all 12 objectives, and each 
form contained the same number and type of items for any jingle objective from 
among the twelve. The pretest, Form X and Form X - Part II, was administered 
at the opening session on the first day of the presession. The post-test, 

Form Z and Form Z - Part II, was administered during the first period on the 
final day. A copy of Form X is included as Appendix A of this report. Form Z 
is available from the authors upon request. 

Table 3 shows the pretest and post-test mean scores by objective and 
presents a group profile for each test. The table reveals that the mean 
pretest score for all participants on the 190-item test was 112.6 (59$) and 
the mean post-test score was i55.7 (82$). Performance gains by objective 
from pretest to post-test ranged from 8$ (objective 2) to 73$ (objective 12). 

As indicated in the table, the participants attained a group post -instructional 
performance level of above 80$ on 9 of the 12 objectives. 

The presession critique forms prepared for use at all 1968 AERA 
presessions and completed by the participants and staff at the conclusion of 
the presession served as additional sources of evaluation. The responses of 
all participants who completed the Participants Evaluation Form are summarized 
in Appendix B. Staff responses are also shown in Appendix B on the form 
entitled Presession Critique for Staff Members. 

The summarized responses of both participants and staff members show 
consistently favorable evaluation of all aspects of the presession. Tabulation 
of positive and negative responses and comments on the Participants Evaluation 
Form, with items related to the hotel facility omitted, reveal a ration of 10 
positive responses from participants to each negative response. 

Based upon the post-instructional achievement of the participants and 
the written evaluations from both participants and staff members, it appears 
reasonable to conclude that the presession was a success. 



o 
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Table 3 

*GROUP PROFILE: MEAN PRETEST AND POST-TEST SCORES BY OBJECTIVE 



Pretest (Form X) 
Post-test (Form Z) 



Obj. Exam No. No. 1 

No. Content Section Items Right Percent Correct 

Pre Post 0 10 20 30 40 50 60 *70 80 90 




* 
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PRESESSION CRITIQUE FOR STAFF MEMBERS 
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Indicate your observation and judgment by checking each item in one 
column at the left. Items not applicable or not subject to your ob- 
servation should be omitted. Be frank. 

1. Environmental conditions 



a. 

b. 

c. 

d. 

(2 none)e. 

f. 



Classroom spaces 
Work spaces 
Living quarters 

Teaching equipment, aids (chalk boards, public address system, 
Resource material, library 
Eating facilities 



2. Participants 

a. Appropriateness of academic backgrounds 

b. Sufficiency of research experience 

c. Willingness to work 

d. Intellectual curiosity 

e. Concern for applicability of techniques 

f. Aspiration 

g. Immediate preparation for Presession 
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3. 



b. 

c. 

d. 

(1 none)e. 

f. 



Organization 

a. Adequacy of notice to prospective applicants 
Sufficiency of preplanning 
Smoothness of operation 
Adaptability to obstacles and feedback 
Sensitivity to grievances 
Adequacy of financial support 



4. Schedule 

a. Appropriateness of five days for the job 

b. Time spent efficiently 

c. Events sequenced appropriately 

d. Punctuality 

e. Balance between formal, informal affairs 

f. Quantity of discussions 

g. Quality of discussions 

h. Quality of formal presentations 

i. Unobtrusiveness of evaluation efforts 

j. Methocs of evaluation 
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__ a. 

_ __ b. 

2 L (1 none)c. 

3 d. 



5. Outcomes 

Intended content was actually taught 
Increase in participant understanding 
Improvement in attitude toward research 
Personal associations initiated , 



etc. 
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6. In general was the Presession well organized? yes - 5 



7. Were the facilities suitable for the activities which you had planned? 
If not, specify. 



yes - 4 Adequate - 1 



8. Should Presessions be limited to the same hotel, or the same city, in 
which the annual meetings will be held? , 

yes - 4 Depends - 1 

9. Were you to do the same assignment over, in what major ways, if any, 
would you change your contribution? More structured handouts . i 

None - 2 Be less directive - 1 (1 no answer) 



10. Do you wish that the Director had made firmer arrangements to assure 

participants and you of the staff opportunity to meet in pairs or small 



groups i 



no - 5 



11. Were the objectives you set for yourself during the Presession attained? 

yes - 4 (1 no answer) 

12. Are you inclined to urge your colleagues to become staff members for 
such an institute or Presession? 

yes - 5 

13. In what ways, if any, did you as a staff member benefit personally as a 
result of your participation in this Presession? 



Increased frustration tolerance for colleagues ~ 1 
Feedback on instruction - 1 

Realize necessity for justifying dogma underlying 
> product research and development - 1 

Became, acquainted with several people with similar 
professional interests - 1 
(1 no answer) 
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PARTICIPANT EVALUATION FORM 



Directions : Please respond with a word, a phrase, or one or more sentences to as many 

of the following questions as you can. Your frank and honest evaluation can only benefit 
everyone concerned. Do not identify yourself by name unless you prefer to do so. 

Environment and Facilitie s 



1 . 



a. 



To what extent did 
journals interfere 
this session? 

None - 20 



th* '■ Ive availability or vr/ reliability of books and 
v:. -omote your •• •-?ster the content of 

None because of availability of *- ,«». clouts “ ^ 



b. To what ex ten*- ’ reproduced materials given to you by the staff improve matters 

Very pos • 21 Positive - 7 Neutral - .. 

2. a. Did you feel t^- a. "place to work. " el^'^er alone or in small groups? 

No - 30 Ye ^ - •*. 

* • 

b. Was your room satisfactory? yes - 25 No - 4 Neutral - 5 



3. a. Which features of the meeting rooms were inadequate or not conducive to learning? 
Crowded - 23 Too hot - 2 None - 2 

b. Which features were especially facilitative in the same regard? 

None - 5 Tables - 4 

Scheduling and Organization 



4. a. Was five days too long a period to leave your work at home for the purpose of 
attending this session? 

No - 19 Yes - 8 

b. Was five days too short a period in which to learn much of the content of this 
session? No _ 2 8 Yes - 2 



5. a. Were you allowed enough time in which to pursue activities of your own choosing? 

Yes - 33 No - 1 

b. Would you have preferred not to meet in the evening after dinner? 

,.;,m H No - 23 Yes - 4 Neutral - 5 

c. Would you have preferred more' or fewer meetings per day than there actually 
were? Or was the number of meetings per day agreeable to you? 

Agreeable - 29 More - 1 Fewer - 1 

6. a. Were the individual lectures too long to sit and listen or take notes? 

No - 29 Yes - 2 Sometimes - 2 

b. Were the lectures scheduled in an appropriate sequence? 

Yes - 31 No - 2 

7. Did you have sufficient opportunities to interact with other participants? 

Yes - 32 No - 3 



8. a. Were the instructors too inaccessible or unapproachable so that you did not 
get the Individual attention that you desired? 

No - 31 Somewhat - 4 

b. Was it helpful to have graduate student assistants present? 



Yes - 24 Neutral - 6 
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9. Did the attempts to evaluate your progress and reactions during the session 
(and at this moment) interfere with your work here? 

No - 29 Yes - 2 

10. In general, was the Presession well organized? Yes . 2 8 Could be improved 

Content and Presentation 



11 . 



3 • 



Did the content of the lectures and readings presuppose far more previous 
training than you had? No . 27 Less . 2 To some extent . 2 



b. Should less training in these areas or more have been presupposed? 
No - 21 More training - 8 



12. To what extent was the content of the lectures and readings relevant to what 
you hoped to a*-' omplish during the session? 4 

Very relevant - 19 Relevant - 4 Not relevant - 3 Hp* 

13. a. Were the lecturers stimulating and interesting? 

Yes - 16 Generally - 6 Varied with individuals - 5 No - 1 

b. Were the lecturers competent to speak on the subject assigned them? 

Yes - 28 Fair - 3 

c. Were the lecturers well prepared? 

Ye3 - 24 No - 5 Varied - 4 

14. Were you disappointed in any way with the group of participants? 

No - 21 Yes - 6 



Answer each of the following only by checking the more appropriate blank: 



15. If you had it to do over again would you apply for this Presession which you 
have just completed? Yes 27 No 7 



16. If a presession such as this is held again would you recommend to others like 
you that they attend? Yes 27 No 5 



17. Do you anticipate maintaining some sort of contact wic'n .r least one of the 
Presession staff? Yes 23 No 6 



18. Do you feel that AER-- is making an important contribution to ^cation by 
sponsoring presessions suer* as this one? Yes 31 N o 1 

19. Do you feel that any torn ..as Uc*ppened during cnese five day- to make it more 

likely that you will you-, present position of employment? Ye s 4 No 27 

20. Is it likely that you Wi.il collaborate in research wit., someone else attending 
this Presession (other than those you already were likely to collaborate with)? 
Yes 15 No 15 



21 . 



Do you feel 
during this 



that the 
five-day 



staff should feel that it 
presession? Yes 29 No 2 



has accomplished 



its objectives 



i 

I 
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APPENDIX A 



CRITERION TESTS 



EXAMINATION: THE DEVELOPMENT OF INSTRUCTIONAL PRODUCTS 

FORM X 



Southwest Regional Laboratory for Educational Research and Development 

11300 La Cienega Boulevard 
Inglewood, California 

1967 



Form X and Form X - Part II, presented in this appendix, served as the 
pretest. The post-test p Form Z and Form L - Part II, are available from 
the authors upon request. 



Examination: The Development of Instructional Products 

Form X 

* 

W. James Popham 



General Directions: Complete only the sub-tests below which are checked , 
as directed in the examination booklet. You may write on the examination 
booklet itself, but please make all of your responses on the answer sheet 
which has been provided. Be sure to write your name on the answer sheet. 



I. 

II. 

III. 

IV. 

V. 

VI. 

VII. 

VIII. 

IX. 

X. 

XI. 



THE DEVELOPMENT PROCESS 
INSTRUCTIONAL OBJECTIVES 
OBJECTIVE DOMAINS 
PERFORMANCE STANDARDS 
CRITERION MEASURES 
PROTOTYPE TEST ITEMS 
INSTRUCTIONAL SPECIFICATIONS 
APPROPRIATE PRACTICE 
KNOWLEDGE OF RESULTS 
PROMOTING LEARNER INTEREST 
AVOIDING IRRELEVANCIES 



PART I. THE DEVELOPMENT PROCESS 

Part I. Directions : This part of the examination consists of five brief 

descriptions of segments in the product development process. Each descrip- 
tion is followed by ten statements, some of which are correct. You are to * 
read each description, then on the answer sheet which has been provided, 
mark an A for each statement which is correct . Nothing needs to be marked 
for a statement which is incorrect. If Insufficient information has been 
presented for you to judge the correctness of a statement, leave the item blank. 

Generally speaking, certain incidents in the development of instructional 
products are recounted in the fictitious descriptions. Your task is to 
identify correct procedures which were employed or errors which were 
made by the product developers. No attempt has been made to be devious 
in the examination. You need not "read between the lines" in order to 
judge the correctness of the 50 statements. The answers should be apparent 
to you if you are familiar with appropriate steps in the product development 
process. Be sure to use the answer sheet for your responses. Now commence 
with the first description and its accompanying ten statements. 



k 

Parts IV and VI were prepared by Eva L. Baker. Part VII was prepared 
by Robert J. Berger 
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Exercise One 



It is mid-December and Frieda, a regional laboratory employee, has been 
given the responsibility of developing a short self-instruction program to teach 
sixth grade pupils how to use commas. She has received a set of five explicit 
instructional objectives from the individuals who originally formulated the 
project. Along with these objectives there are samples of a single prototype 
tesL items for each objective. Frieda has been told that the instructional 
product is to take no more than four hours of the average learner's time. 



The first thing she does is to develop a 40-item criterion test to be used 
at the close of the program. She has each of the five objectives represented 
bv at least five test items, although two objectives which she feels to be more 
important are represented by 10 items. 



Frieda carefully considers the enroute (intermediate) behaviors which the 
learner must master on his way to the criterion behaviors and then sequences 
these from least to most difficult, ending with behaviors equivalent to those 
called for in the instructional objectives. She then prepares practice sequences 
for each of the enroute and terminal behaviors so that the learner will be able 
to practice using commas in a variety of situations. After having three colleagues 
react to her first version of the instructional product, she makes a number of 
revisions . 



Frieda then arranges to field test the program in the public schools and 
secures the cooperation of a nearby elementary school. She arranges to use 
three classes for approximately one week and administers the program in early 
February. At its conclusion the 40-item criterion test is given to each of 
the 86 children who completed the program. 

Frieda is pleased that all youngsters were able to finish the instructional 
product in three hours or less. She is somewhat concerned, however, that the 
average score on her 40-item test is only 21.2 correct. She resolves to study 
the post-test results as well as the responses made by pupils during the program 
and to make the revisions which seem dictated by the data. 

********** 



Which, if any, of the following statements are correct? (Mark on the 
answer sheet an A for any which are correct.) 

1. Frieda was probably correct in providing practice behaviors for 
the learner which were equivelent to those called for in the 
instructional objectives. 

2. She should have required at least two prototype test items along 
with the instructional objectives. 

3. Each objective should have been represented equally on the criterion 
test. 

4. Frieda field-tested her first version of the product on too many 
learners. 

5. She should have pre-tested the subjects. 

6. Her program was dull. 



7 



• Too much time was taken to complete the first version of the product. 

8. Five objectives are too many for such a program. 

9. The enroute behaviors should not have been sequenced from least 
to most difficult. 

10. More than three colleagues should have reacted to the first version 
of the program. 



Exercise Two 



A group of three beginning product developers has been assigned the general 
task of preparing a 50-minute self-instruction sequence for learners in the 
field of English. The resulting product will be used by junior high school 
instructors, more specifically, ninth grade English teachers, as a remedial 
program for students who are having difficulty with one or more topics in the 
subject. 

After some discussion (approximately three days) among themselves and a 
few experienced instructional programmers, the three decide to develop a 
remedial program which will increase the student's ability to diagram standard 
sentences so that each of the eight parts of speech are clearly identified. 

They then turn to the task of deciding on an appropriate criterion test, 
spending the next month in a series of carefully organized discussions re- 
garding the possible methods of assessing a learner's ability to manifest 
mastery of the diagramming process. For example, during one meeting the three 
individuals develop sample items and take turns answering them, then analyzing 
the adequacy of each other's responses. The following objective is agreed upon 
for the program: 

When presented with 10 previously unseen sentences, five simple (that is, 
one clause) and five complex (that is, two or more clauses), the learner will 
be able to diagram at least eight without any errors according to the procedure 
specified in the 50-minute instructional program. Ninety per cent (or more) 
of the learners who complete the program must satisfactorily achieve this level 
of proficiency. 

The terminal behavior having been selected, the three product developers 
then prepare 25 test items based on the above objective and arrange to administer 
the 25-item test to a group of 30 ninth grade English students. Gratifyingly , 
the students are not able to perform well on the test, so the three developers 
next address themselves to a careful task analysis in order to identify ne- 
cessary types of entry behaviors (the skills the pupil possesses before starting 
the program) and enroute behaviors (the intermediate skills the learner must 
achieve in order to attain the terminal behavior). Having done this, approxi- 
mately four days being expended on the task, an appropriate pre-test is pre- 
pared including items which reflect desired (1) entry behaviors, (2) enrout:. 
behaviors, and (3) criterion behaviors. This test is then administered to ten 
pupils and, having carefully analyzed their responses, the product developers 
begin to prepare the first draft of their instructional product • . . 



********** 
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; Which, if any, of the following statements are correct? 

11. The preparation of the criterion test quite properly preceded 
the development of the instructional product. 

12. It was unnecessary to assess learners' ability to perform entry 
and enroute behaviors. 

13. Too much time was spent in the determination of a suitable 

instructional objective. i 

14. The instructional objective decided on was not sufficiently precise. 

15. The minimum level of proficiency expected of learners was not 
well specified. 

16. The adequacy of the original formulation (i.e., the selection of 
the particular topic) was not well justified. 

17. Too much time was spent by the product developers on the original 
selection of a topic. 

18. The expanded pre-test, the one with the addition of items based 
on entry and enroute behaviors, should have been given to at 
least 30 learners. 

19. The first version of the instructional sequence should have been 
prepared and tried out prior to the administration of the pre-test. 

20. It was quite appropriate for the product developers to undertake 
the task analysis of entry and enroute behaviors. 



Exercise Three 

Fred Peabody, an experienced instructional programmer, has been assigned 
the responsibility of developing a self-instruction program to teach third grade 
youngsters how to add and subtract fractions. One of the first things he does is 
to go to a third grade classroom and ask permission to talk with six of the 
youngsters. Individually he discusses the topic of fractions with each of them 
and attempts to find out what the students already know that will be relevant to 
his task. Having completed his assessment of the data secured from these inter- 
views, he devises the following objective for his program: "At the conclusion 
of the program the learner will manifest a sophisticated ability to handle addi- 
tion and subtraction problems involving fractions." 

After consultation with the teachers and the administrators he prepares a 
formulation paper in which he identifies the above objective and attempts to 
support the value of the proposed program. He cites the opinions of teachers 
who indicate that a short-term program such as that which he proposes will have 
considerable utility in their classes and cites evidence from published bibli- 
ographies of program material that there is currently no such short-term program 
available to teachers in a form which does not require the use of teaching 
machines. Accordingly, he proposes that his program will take approximately 
one and a half or two hours to complete and will be presented by printed-paper 
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booklets. This formulation paper is presented to a number of colleagues with 
whom Peabody works and, having read it, they agree that his selections of topic 
and presentation medium are sound. 

He develops a criterion test in which the learners must add and subtract 
pairs of fractions. He also prepares other items which assess the learners' 
ability to add and subtract whole numbers. He combines these to form a pre- 
test which he administers to a sample of 25 third graders drawn from a nearby 
elementary school. 

Having determined that the youngsters cannot perform the terminal behavior, 
but do possess the desired prerequisite skills, Mr. Peabody then prepares an 
early version of his program. 

Since he employs a variety of approaches in his programming efforts, Mr. 
Peabody attempts to select a tactic for this particular product. After intro- 
ductory remarks and a certain amount of exposition (approximately three pages' 
worth) .he gives the student a series of ten sets of simple fractions to add. 

After they have concluded adding this set, he provides them with the correct 
answers which they can see by turning a page and comparing the proper answers 
with the answer sheet on which they have been instructed to make the^r responses. 
Several more sets of these addition exercises are provided in order that the 
learner may practice the operation of fraction addition. After each ten problems 
Mr. Peabody provides knowledge of results in a manner similar to that described 
above. 

For variety's sake, however, when it comes to subtraction of fractions, he 
approaches it in an entirely different fashion. He presents a series of short 
story (word) problems to the learner which involve the subtraction of fractional 
quantities. The learner is obliged to "think through" what fractional quantities 
are required and then form a mental subtractional operation regarding each of. 
these particular problems. After each set of three word problems, once more 
knowledge of results is provided whereby the learner can check the accuracy of 
his responses. In all, five sets of these subtraction word problems are pre- 
sented. 

Mr. Peabody has multiple copies of his program prepared on a mimeograph 
machine and takes it to the same elementary school where he earlier secured 
such excellent cooperation. He administers the program at the school where 
he had his original interviews, using again the same students that were so 
helpful two weeks earlier and two other classes as well. At the conclusion 
of the program which, as he predicted, took approximately an hour and 45 minutes 
for most students to complete, he administers the series of 20 simple addition 
problems and 20 simple subtraction problems, each involving two fractions. On 
the 40-item test he is disappointed when the mean performance of the 28 pupils 
who complete the program pre-test is only 26.3. Mr. Peabody decides to revise 
his program consistent with these data. He is particularly anxious to check 
the responses which learners made during the program since he believes these 
will provide him with clues as to what sections are most in need of alteration. 
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Which, if any, of the following statements are correct? 

21. Mr. Peabody wisely planned to use learner responses during the 
instructional program as a guide for improving his program. 

22. The performance of Mr. Peabody's students on his program was 
acceptable and he ought to be satisfied. 

23. Mr. Peabody did not define the terminal behavior adequately in 
his objective. 

24. Too much delay may have been involved in the knowledge of results 
provided for the majority of the learners' responses. 

25. The programmer should not be concerned with "variety" in 
developing his instructional materials. 

26. Mr. Peabody should have involved more learners during h:s initial 
exploratory interviews. 

27. He probably field tested his first version of the program with 
too many learners. 

28. He should have field tested his program with a group of students 
other than those involved in his initial exploratory interviews. 

29. The operations associated with his formulation procedure were 
inadequate. 

30. Peabody had too few items on his criterion test. 



F. x ercise Four 

Mrs. Shear has acquired a reputation in the past several years of being 
a remarkably skilled product developer. She believes that much of her reputation 
is due to the fact that she has developed a workable procedure for accomplishing 
instructional objectives. She has discovered that the preparation of audiotape 
narration, coupled with the use of visual transparencies placed on an overhead 
projector by the teacher, efficiently accomplishes the behavior changes she 
desires. Furthermore, she always tests her program through the use of five 
alternative multiple choice examinations which she has become most adept at 
constructing. No matter what the objective, whether cognitive, affective, or 
psychomotor in nature, and no matter how complex, Mrs. Shear analyzes it in 
such a way that it can be handled through the use of this audio tape-trans- 
parency approach. 

The first thing she always does is construct a relevant multiple choice 
test. She uses five choice items because of their greater efficiency in dis- 
criminating between the more and less knowledgeable learners.. She is careful 
to try out her tests with an appropriate group of learners so that she can tell 
which items properly discriminate between the more and less knowledgeable 
learners in the group. Although sometimes her methods fall short of expectations, 
Mrs. Shear's reputation as a productive programmer is widely held among her 
colleagues. 

************ 



Which, if any, of the following statements are correct? 

31. Mrs. Shear quite appropriately prepares her criterion test before 
developing the instructional materials. 

32. Mrs. Shear is too inflexible regarding her selection of programming 
strategies. 

33. Her approach is bound to fail with highly creative youngsters. 

34. Few teachers would be willing to place transparencies on an 
overhead projector as instructed in the program. 

35. There are some criterion behaviors for which multiple choice tests 
are inappropriate, hence, Mrs. Shear ought not to use them in all 
situations . 

36. Mrs. Shear's tests are inappropriate because they are built on 
a norm referenced (comparing student to student) rather than a 
criterion referenced basis (comparing student performance to goals). 

37. Mrs. Shear's programs are probably uninteresting. 

38. She ought to achieve her objectives in every situation she attempts, 
probably on the first or second draft of her program, or she does 
not deserve the reputation she has as a skilled programmer. 

39. Mrs. Shear should realize that group-paced programs have little 
place in the public schools. 

40. Her programming approach will prove effective only in the cognitive 
domain. 



Exercise Five 



Mr. Smith has been assigned the task of developing a one-week (approxi- 
mately five 50-minute periods) group-paced instructional program designed to 
teach high school chemistry students to treat correctly certain analytic equations 
and problems involving unknown chemical elements. Mr. Smith does not attend to 
the formulation process because this has been done by others. His responsibility 
is to develop the actual instructional material. The instructional objective 
which has been given to him by members of the formulation team is the following; 

At the conclusion of the instruction at least 80 per cent 
of the learners will be able to solve seven of ten equation 
problems involving an unknown compound. 

Mr. Smith arranges to talk to several high school chemistry instructors 
and a half a dozen high school chemistry students to’ secure some ideas as to the 
proper tactics to employ in teaching the particular subject. The students are 
asked how much they already know of the topic so that Mr. Smith can identify the 
competencies he can build upon when preparing the instructional sequence. 
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He develops four programs, each lasting approximately 40 minutes, with 
the expectation that these early versions will be revised and augmented. He 
adopts a "lean" strategy m programming in which he offers the minimum amount 
of instructional materials that, he believes is requisite, anticipating that if 
the program fails it will be easier to add to it than to subtract superfluous 
material from an effective program sequence. All of the programmed material is 
transferred to an audiotape so that it can be coordinated with visual materials 
which are presented on 2 x 2 slides, he ultimately plans to transfer the visual 
sections to a filmstrip, but believes the slide presentation will offer more 
flexibility for the subsequent addition or deletion of modified visuals. 

He next develops a criterion test consisting of thirty items in which the 
student is presented with verbal descriptions of chemical interactions and asked 
to describe with chemical equations the nature of the quantitative equations which 
have been verbally described. 

He tries the program with four learners who answer, respectively, 21, 25, 

26, and 27 items correctly on the 30-item test. 

*************** 

Which, if any, of the following statements are correct? 

41. Mr. Smith's expectation that the early version of the program 
will be revised is realistic. 

42. His students did not perform as well as they should have on his 
first draft materials. 

43. A "lean" programming strategy has been demonstrated to be 
ineffectual in this type of task. 

44. Mr. Smith should have prepared his criterion test prior to the 
development of his first version instructional product. 

45. Mr. Smith's instructional materials were probably dull. 

46. The use of audiotape and filmstrip is inconsistent with the 
notion of group-paced programs. 

47. Mr. Smith's criterion test was not appropriate for the 
instructional objectives he had been given. 

48. Mr. Smith, or any programmer, has the clear responsibility for 
evaluating the adequacy of the formulation operation no matter 
at what point he is introduced to the development process. 

49. Mr. Smith should net have consulted teachers and pupils prior 

to the development of his first draft of instructional materials, 

50. Mr. Smith should have developed first draft materials which were 
exactly as long as those meant for the program. 
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PART II. INSTRUCTIONAL OBJECTIVES 



Part II Directions ; In the following list identify any properly stated instruc- 
tional objectives by marking an A on the answer sheet. Nothing need be marked 
for improperly stated objectives. 



51. 

52. 

53. 

54. 

55. 

56. 

57. 

58. 

59. 

60. 



grasp the significance of the Treaty of Versailles, 
have an attitude favorable to English grammar indicated 
to a questionnaire, 
know six verbs. 

learn the names of the common tools in wood shop, 
list three major causes of the Civil W cv on the 

know the important battles in World War I. 
prefer cooking to sewing. 

be able to correctly thread a sewing machine, 
pay attention as the teacher demonstrates the use of the 

lathe. | # 

The student will be able to develop a sense of the cultural unity of man. 



The student will 
The student will 
by his response 
The student will 
The student will 
The teacher will 
chalkboard. 

The student will 
The student will 
The student will 
The student will 



PART III. OBJECTIVE DOMAINS 



Part HI Directions ; Classify each objective below by marking the correct 
letter according to the following scheme; j 



A. psychomotor 

B. affective 

C* cognitive- -higher than lowest level 
D. cognitive-- lowest level 

The learner; 

i 

61. is able to choose the best of two solutions to a geometry problem using 
standards given by the teacher. 

62. exhibits tolerance for others by displaying good manners toward those of 

minority groups. # 

63. lists the names and contributions of five key curriculum workers. 

64. properly knits a baby blanket. 

65. scores well on the Minnesota Teacher Attitude Inventory. 

66. uses instructional principles properly in planning daily lessons. 

67. plays table tennis according to rules well enough to beat three inexperienced 

girls 100% of the time. 

68. correctly recites Gettysburg Address from memory. 

69. scores 80% or better on a spelling quiz. 

70. displays interest in higher mathematics by volitionally attending lectures 

on this topic j 
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PART IV, PERFORMANCE STANDARDS 

Part IV Directions : For the following objectives, mark A if the objective has 

only a student minimal level of learner behavior; B if the objective has a 

class minimal level of behavior; and C if the objective has no minimal level 

of learner behavior. 

71. The class will answer correctly 10 out of 12 multiple choice questions 
on the Roman Empire. 

72 i The students will compose an essay on the topic of their summer 
vacation. 

73. At least 10 students in the class will sign up for a senior life saving 
course at the conclusion of a unit on water safety. 

74. Seventy- five percent of the students will understand differential 
equations. 

75. Students will recite with no more than one error Milton's sonnet 
"On His Blindness". 

76. 60% of the students will prepare 500 word book reports on famous 
social scientists. 

77. The students will thoroughly comprehend at least 80% of the scientific 
theories treated in class. 

78. The students will paint a still-life study employing two point 
perspective and at least three colors. 

79. Everyone in class will orally recite a given Spanish dialog with no 
errors in pronunciation. 

80. Students will be able to match chemical compounds with their valences 
on a written test. 



PART V. CRITERION MEASURES 

Part V (a) Directions: For the following list of five items distinguish 

between those which could be employed as educational criterion measures and 
those which could not by marking an A on your answer sheet for each item 
which could be used as a criterion to evaluate educational programs. Be 
careful to match the item numbers with the appropriate item on the answer 
sheet. 

81. Standardized achievement tests 

82. Locally constructed tests of pupils' progress in spelling 

83. The age of learners 

84. An anonymous self-report questionnaire which, among other more obvious 
purposes, contains a question soliciting the respondent's values 
regarding labor unions 

85. The number of serious pencil marks, carvings, etc., on pu^il desks 
which must be removed by maintenance personnel each summer 
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Part V (b) Directions : Using the four category scheme presented below, classify 

each of the following five lettered items by selecting the appropriate letter 
in the space for each item on the answer sheet. 



Classes of Criterion Measures 

A. Learner-behavior--natural conditions 

B. Learner-behavior- -manipulated conditions 

C. Behavicr-product--natural conditions 

D. Behavior-product--manipulated conditions 

86. Scores on the Kuder test of vocational interest 

87. Instances of pupil misbehavior during recess periods 

88. Surreptitious observations of learner behavior in nationally distributed 
"situational stress" tests involving accomplices 

89. "Courtesy" as reflected by adolescent boys giving their seats on the 
bus to women who might otherwise be obliged to stand 

90. Final extemporaneous speeches in senior English class 



PART VI. PROTOTYPE TEST ITEMS 

Part VI (a) Directions : Mark an A on the answer sheet by the number of 

any objective which includes a statement of presentation conditions. 

91. The learner will compose in writing a four line verse. 

92. The student will compare Romanticism <md Victorianism literary movements. 

93. The teacher will list five elements necessary in a particular geometry 
proof. 

94. Given pictures of four colored objects, the child will circle the object 
which is red. 

Part VI (b) Directions : For the following pair of objectives and items, 

mark A if the item corresponds to the objective in terms of response called 
for, directions , and presentation conditions . If the item does not correspond, 
with the objective, mark B, For each objective and item you should make 
three responses on the answer sheet. 

***** 

Objective: The student will write the course of action most consistent 

with the tenets of good citizenship outlined in class when given a 
social problem not previously encountered. 



Item : Choose a social problem you are familiar with and in less than 

300 written words describe how you would deal with it in terms of the 
citizenship concepts described in class. 

***** 

Student Response (95) Directions (96) Presentation Conditions (97) 
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it 



it 



* 



it 



* 



_Ob.iecti.ve : The student will be able to select all written notations 
which describe permissible moves in a chess game when presented with 
four choices. 

item: For the opening move by white, which of the following moves 



are permissible? 


(Check your 


answers.) 




(a) K-KZ _ 


(b) P-QN3 


(c) N-QB3 


(d) QB-R5 


* 


* 


* * 


• 

* 



Student Response (98) Directions (99) Presentation Conditions (100) 

PART VII. INSTRUCTIONAL SPECIFICATIONS 

Part VII (a) Directions : Indicate which component of the IS is most nearly 

described or identified by each of the following items by marking on your 
answer sheet: 

A. for "Terminal Behavior" 

B. for "Instructional Cue" 

C. for "Elicitor" 

D. for "Limits" 

E. for "Entering Behavior" 

101. Provides rules or procedures designed to improve learner performance. 

102. Contains prototype criterion items. 

1C3. Describes the stimulus conditions under which the criterion behavior 
will be demonstrated. 

104. Contains the information the learner requires to perform the criterion 
behavior. 

Part VII (b) Directions : This portion of the examination consists of two 

terminal behavior is followed by potential components of the instructional 
specification. For each statement which is appropriate to its terminal 
behavior, whether it is a complete component or not, mark an "A" on your 
answer sheet. Nothing need be marked if the statement is not appropriate 
to the terminal behavior. 

No attempt has been made to be devious or to trick you with tests of your 
knowledge of subject matter pertaining to each objective. You should only 
concern yourself with the appropriateness of the component to the terminal 
behavior, not authenticity of the content. 

***** 

"To construct an equilateral triangle, given the necessary equipment." 

105. Instructional Cue: "An equilateral triangle is a triangle in which 

all sides and angles are equal." 

106. Elicitor: "Name an object that has equal sides and equal angles." 

107. Limits (negative): "Triangles with less than 3 equal sides and 3 
equal angles." 



O 
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108. Entering Behavior: "To identify triangles, given examples of triangles, 

squares, and rectangles." 

***** 

"To write the longitude and latitude of any given position on a map, 
given a map with longitude and latitude lines." 

109. Instructional Cue: "Longitudes and latitudes are essential for 

locating positions on the open sea." 

110. Elicitor: "Degrees latitude on the position indicated are , 

degrees longitude are . " 



PART VIII. APPROPRIATE PRACTICE 

IX. KNOWLEDGE OF RESULTS 

X. PROMOTING LEARNER INTEREST 

XI. AVOIDING IRRELEVANCIES 

Parts VIII, IX, X, and XI. Directions: First, read the short self-instruction 

program dealing with statistics. (You may wish to respond to the program as 
though you were the intended learner.) Then answer the series of examination 
questions 



MADAM, MAY 1 PLEASE MEASURE YOUR CENTRAL TENDENCIES? 

(A Non-Exemplary Program) 

by 

W. James Popham 





Program Objective : At the conclusion of the program the learner will 

be able to select the correct numerical values of th? mean, median, or mode 
from multiple choice alternatives when presented with a set of fictitious data. 
Enroute Objective : The learner will be able to match the terms mean, 

median, and mode with definitions of these measures. 

Enroute Objective : The learner will be able to compute the numerical value of 

the mean, median and moue from small sets of fictitious data. 

Prerequisite Behaviors ; As necessary entry behaviors, the learners 
should be able to read and perform the following operations with one, 
two, and three digit numbers: add, subtract, multiply, divide. 
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A* As an individual concerned with the development of instructional 
materials , you will encounter situations in which you wish to describe 
how well a group of learners performed on a criterion test after com- 
pleting your instructional sequences. Ideally, this description should 
be supplied as parsimoniously as possible to save your time as well as 
those with whom you are attempting to communicate. One of the most ef- 
ficient ways to describe a set of data is through the use of statistical- 
measures of central tendency. 



B. Statistical measures of central tendency are numerical indicators 

of the manner in which the scores of individuals in a group of scores 

(such as test data) tend to cluster near the center of the scale on which 
the scores are measured. The three measures to be treated in this 

program are the mean , the median, and the mode. 



C, The mean is calculated by adding together all of the scores in a 
set of scores (also called a distribution) and then dividing them by 
the number of scores in the set. For example, consider the following 
set of seven scores: (8, 7, 5, 4, 3, 1, 0). When these scores are 
added together, their sum is 28. Dividing 28 by 7, the number of scores, 
we find that 4 is the mean. For the following set of four scores (10, 

8, 8, 2) then, we can see that 7 is: 

a. the mean c. the mode 

b. the median d. none of the above 

For these and subsequent items in which you are to make a choice or 
supply an answer, please make your response directly on the booklet, 
then read beyond the three asterisks. It may be necessary to mark off 
the section below the asterisks so that you do not inadvertently see 
the correct answer. For the question above, circle the letter of the 



best answer. 
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Since 7 is derived by adding all scores (28) and then dividing by the 
number of scores (4), it is the mean. 



D# The median is the point which divides a set of scores into two equal 
halves. For instance, in the following set of scores (9, 9, 6, 5, 2) 
the number 6 splits the scores into two equal halves, hence it is the 
median. 



E. Sometimes the median is not an actual score. See if you can determine 
the median for the following set of scores. This is a difficult problem 
and if you get it right, you will be one of the eleven per cent who answer 
correctly. See if you can choose the right answer and avoid the 89% 

I 

stigma! 

Scores: 7, 7, 6, 4, 2, 1 
Median choices (circle one): 

a. 6 

b. 5.5 

c. 5 

d. 4.5 

* ie * 

If you chose answer C you are a member of the "lofty eleven." To calculate 
the midpoint of this distribution, it would be necessary to interpolate 
between scores of 6 and 4 to obtain a 5. 



F. The mode is the most frequently appearing score in a distribution. Thus 
in the following set of scores (9, 8, 4, 2, 2, 1) the score 2 appears most 
frequently so it is the mode. What would the mode be for this set of scores 
(8, 8, 7, 7, 7, 15? 

a. 8 

b. 7 

o c. 2 

cKJl 



